Planet Python

Last update: March 22, 2019 07:47 AM UTC

March 21, 2019

Test and Code

69: The Pragmatic Programmer - Andy Hunt

Andy Hunt and Dave Thomas wrote the seminal software development book, The Pragmatic Programmer. Together they founded The Pragmatic Programmers and are well known as founders of the agile movement and authors of the Agile Manifesto. They founded the Pragmatic Bookshelf publishing business in 2003.

The Pragmatic Bookshelf published it's most important book, in my opinion, in 2017 with the first pytest book available from any publisher.

Topics:

• The Pragmatic Programmer, the book
• The Manifesto for Agile Software Development
• Agile methodologies and lightweight methods
• Some issues with "Agile" as it is now.
• The GROWS Method
• Pragmatic Bookshelf, the publishing company
• How Pragmatic Bookshelf is different, and what it's like to be an author with them.
• Reading and writing sci-fi novels, including Conglommora, Andy's novels.
• Playing music.

Special Guest: Andy Hunt.

Sponsored By:

• PyCharm Professional: Try PyCharm Pro for an extended 4 month trial before deciding which version you need. If you value your time, you owe it to yourself to try PyCharm. Promo Code: TESTNCODE2019

Support Test & Code - Software Testing, Development, Python

March 21, 2019 05:00 PM UTC

Caktus Consulting Group

Coding for Time Zones & Daylight Saving Time — Oh, the Horror

March 21, 2019 04:45 PM UTC

PyCharm

Deep-dive tutorial series on React, TypeScript, and TDD

React is obviously super-hot, TypeScript has recently become super-hot, and the combination is attracting increasing interest. Combined with Jest for test-driven development (TDD)…great developer experience.

Want to see what our IDE has to offer for fullstack Python projects which use React frontends? We’re proud to release a long, LONG “under construction” tutorial series: React, TypeScript, and TDD. Videos, code, write-ups with screenshots, organized into 12 parts.

And along the way, a formal unveiling of the PyCharm Guide, a compendium of resources for becoming a PyCharm badass.

The Tutorial Series

It’s hard to find quality, updated resources for learning React and TypeScript with Jest testing. In fact, that was the genesis of this series. I wanted to remember what I (painfully) learned, so I started documenting it.

The idea grew into what we developer advocates at JetBrains do a ton of: videos. And code. And caring deeply about helping our customers get “into the flow” with new technologies. Our IDEs go the extra mile to do the janitorial work and keep you productive at what you’re doing. I wanted to show what PyCharm, via the exceptional WebStorm team, could do.

But I chose to do the series in very different way: TDD. As each skill is shown, rather than portray it through “switch over to your browser, click around”, we stay in the IDE and learn via tests. Meaning, how the code works rather than how it looks. And most of all, you stay “in the flow.”

Each step in the tutorial focuses on something you’d like to learn in React (e.g. Functional Components.) The step provides working code in a repo for the finished step, an extensive “you can’t go wrong” writeup focused on the “why” as much as the “how”, occasional screenshots, and a narrated video at the end to see it all in action.

One other choice: the series is delivered in a new facility we’re calling the PyCharm Guide.

About the PyCharm Guide

Those watching our recent “tip tweets” may have spotted something different. Some of our tweets appear as “cards” with a link to a full page about that tip. A page with…a lot more information than just the tip:

• A full writeup explaining the tip, what problem it is solving, and variations on the tip for different situations
• Links to related tips, along several axes (technology, topic, author)
• Links to external content (e.g. help system articles)
• A short, narrated video about that tip
• Playlists which organize the tips into a certain sequence

The goal of the Guide is simple: help you become a badass Python developer, putting the power our IDE to work to keep you in the coding flow. The Guide is your “guide”.

To get there, the Guide is done in a different way:

• Content is done in an open source GitHub repo as Markdown using an open source static-site-generator (SSG)
• We’ll thus accept issue tickets, PRs, and contributions, all in a normal community fashion
• We’re doing this in a “let’s see what happens” kind of way, learning from the community to decide what’s next
• A focus on what you want to accomplish (React and TypeScript) as much as how PyCharm works

The Guide is a work-in-progress. Lots more tips to do (42 Tips and Tricks), lots more tutorials to write (pytest, type hints), new content types to add, explanation on how to participate, etc.. At the moment we’re using GatsbyJS as an SSG for the “modern web”, though this might go in a different direction. But that’s ok: the software is less important than the concept and content.

So please, enjoy this 12-part tutorial series on React+TS+TDD. We hope it’s useful, and hope the Guide becomes a useful resource for your learning journey.

March 21, 2019 02:58 PM UTC

Dataquest

Tutorial: Introduction to Git and Github

Learn the basics of version control with Git and Github in this introductory tutorial designed to get you up and running fast.

The post Tutorial: Introduction to Git and Github appeared first on Dataquest.

March 21, 2019 02:00 PM UTC

Real Python

Intro to Object-Oriented Programming (OOP) in Python

Object-oriented programming is one of the biggest and most important subjects in all of programming. This series will provide you with a basic conceptual understanding of Object-Oriented Programming so you can take your Python programming skills to the next level.

You’ll learn how to define custom types using classes, and how to instantiate those classes into python objects that can be used throughout your program.

Finally, you’ll learn how classes can subclass one another with brief introduction to inheritance, which allows you to write maintainable and less redundant Python code.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

March 21, 2019 02:00 PM UTC

Stack Abuse

Python for NLP: Vocabulary and Phrase Matching with SpaCy

This is the third article in this series of articles on Python for Natural Language Processing. In the previous article, we saw how Python's NLTK and spaCy libraries can be used to perform simple NLP tasks such as tokenization, stemming and lemmatization. We also saw how to perform parts of speech tagging, named entity recognition and noun-parsing. However, all of these operations are performed on individual words.

In this article, we will move a step further and explore vocabulary and phrase matching using the spaCy library. We will define patterns and then will see which phrases that match the pattern we define. This is similar to defining regular expressions that involve parts of speech.

Rule-Based Matching

The spaCy library comes with Matcher tool that can be used to specify custom rules for phrase matching. The process to use the Matcher tool is pretty straight forward. The first thing you have to do is define the patterns that you want to match. Next, you have to add the patterns to the Matcher tool and finally, you have to apply the Matcher tool to the document that you want to match your rules with. This is best explained with the help of an example.

For rule-based matching, you need to perform the following steps:

Creating Matcher Object

The first step is to create the matcher object:

import spacy  
nlp = spacy.load('en_core_web_sm')

from spacy.matcher import Matcher  
m_tool = Matcher(nlp.vocab)  

Defining Patterns

The next step is to define the patterns that will be used to filter similar phrases. Suppose we want to find the phrases "quick-brown-fox", "quick brown fox", "quickbrownfox" or "quick brownfox". To do so, we need to create the following four patterns:

p1 = [{'LOWER': 'quickbrownfox'}]  
p2 = [{'LOWER': 'quick'}, {'IS_PUNCT': True}, {'LOWER': 'brown'}, {'IS_PUNCT': True}, {'LOWER': 'fox'}]  
p3 = [{'LOWER': 'quick'}, {'LOWER': 'brown'}, {'LOWER': 'fox'}]  
p4 =  [{'LOWER': 'quick'}, {'LOWER': 'brownfox'}]  

In the above script,

• p1 looks for the phrase "quickbrownfox"
• p2 looks for the phrase "quick-brown-fox"
• p3 tries to search for "qucik brown fox"
• p4 looks for the phrase "quick brownfox"

The token attribute LOWER defines that the phrase should be converted into lower case before matching.

Once the patterns are defined, we need to add them to the Matcher object that we created earlier.

m_tool.add('QBF', None, p1, p2, p3, p4)  

Here "QBF" is the name of our matcher. You can give it any name.

Applying Matcher to the Document

We have our matcher ready. The next step is to apply the matcher on a text document and see if we can get any match. Let's first create a simple document:

sentence = nlp(u'The quick-brown-fox jumps over the lazy dog. The quick brown fox eats well. \  
               the quickbrownfox is dead. the dog misses the quick brownfox')

To apply the matcher to a document. The document is needed to be passed as a parameter to the matcher object. The result will be all the ids of the phrases matched in the document, along with their starting and ending positions in the document. Execute the following script:

phrase_matches = m_tool(sentence)  
print(phrase_matches )  

The output of the script above looks like this:

[(12825528024649263697, 1, 6), (12825528024649263697, 13, 16), (12825528024649263697, 21, 22), (12825528024649263697, 29, 31)]

From the output, you can see that four phrases have been matched. The first long number in each output is the id of the phrase matched, the second and third numbers are the starting and ending positions of the phrase.

To actually view the result in a better way, we can iterate through each matched phrase and display its string value. Execute the following script:

for match_id, start, end in phrase_matches:  
    string_id = nlp.vocab.strings[match_id]  
    span = sentence[start:end]                   
    print(match_id, string_id, start, end, span.text)

Output:

12825528024649263697 QBF 1 6 quick-brown-fox  
12825528024649263697 QBF 13 16 quick brown fox  
12825528024649263697 QBF 21 22 quickbrownfox  
12825528024649263697 QBF 29 31 quick brownfox  

From the output, you can see all the matched phrases along with their vocabulary ids and start and ending position.

More Options for Rule-Based Matching

Official documentation from the sPacy library contains details of all the tokens and wildcards that can be used for phrase matching.

For instance, the "*" attribute is defined to search for one or more instances of the token.

Let's write a simple pattern that can identify the phrase "quick--brown--fox" or quick-brown---fox.

Let's first remove the previous matcher QBF.

m_tool.remove('QBF')  

Next, we need to define our new pattern:

p1 = [{'LOWER': 'quick'}, {'IS_PUNCT': True, 'OP':'*'}, {'LOWER': 'brown'}, {'IS_PUNCT': True, 'OP':'*'}, {'LOWER': 'fox'}]  
m_tool.add('QBF', None, p1)  

The pattern p1 will match all the phrases where there are one or more punctuations in the phrase quick brown fox. Let's now define our document for filtering:

sentence = nlp(u'The quick--brown--fox jumps over the  quick-brown---fox')  

You can see our document has two phrases quick--brown--fox and quick-brown---fox, that you should match our pattern. Let's apply our mather to the document and see the results:

phrase_matches = m_tool(sentence)

for match_id, start, end in phrase_matches:  
    string_id = nlp.vocab.strings[match_id]  
    span = sentence[start:end]                   
    print(match_id, string_id, start, end, span.text)

The output of the script above looks like this:

12825528024649263697 QBF 1 6 quick--brown--fox  
12825528024649263697 QBF 10 15 quick-brown---fox  

From the output, you can see that our matcher has successfully matched the two phrases.

Phrase-Based Matching

In the last section, we saw how we can define rules that can be used to identify phrases from the document. In addition to defining rules, we can directly specify the phrases that we are looking for.
This is a more efficient way of phrase matching.

In this section, we will be doing phrase matching inside a Wikipedia article on Artificial intelligence.

Before we see the steps to perform phrase-matching, let's first parse the Wikipedia article that we will be using to perform phrase matching. Execute the following script:

import bs4 as bs  
import urllib.request  
import re  
import nltk

scrapped_data = urllib.request.urlopen('https://en.wikipedia.org/wiki/Artificial_intelligence')  
article = scrapped_data .read()

parsed_article = bs.BeautifulSoup(article,'lxml')

paragraphs = parsed_article.find_all('p')

article_text = ""

for p in paragraphs:  
    article_text += p.text


processed_article = article_text.lower()  
processed_article = re.sub('[^a-zA-Z]', ' ', processed_article )  
processed_article = re.sub(r'\s+', ' ', processed_article)

The script has been explained in detail in my article on Implementing Word2Vec with Gensim Library in Python. You can go and read the article if you want to understand how parsing works in Python.

The processed_article contains the document that we will use for phrase-matching.

The steps to perform phrase matching are quite similar to rule based matching.

Create Phrase Matcher Object

As a first step, you need to create PhraseMatcher object. The following script does that:

import spacy  
nlp = spacy.load('en_core_web_sm')


from spacy.matcher import PhraseMatcher  
phrase_matcher = PhraseMatcher(nlp.vocab)  

Notice in the previous section we created Matcher object. Here, in this case, we are creating PhraseMathcer object.

Create Phrase List

In the second step, you need to create a list of phrases to match and then convert the list to spaCy NLP documents as shown in the following script:

phrases = ['machine learning', 'robots', 'intelligent agents']

patterns = [nlp(text) for text in phrases]  

Finally, you need to add your phrase list to the phrase matcher.

phrase_matcher.add('AI', None, *patterns)  

Here the name of our matcher is AI.

Applying Matcher to the Document

Like rule-based matching, we again need to apply our phrase matcher to the document. However, our parsed article is not in spaCy document format. Therefore, we will convert our article into sPacy document format and will then apply our phrase matcher to the article.

sentence = nlp (processed_article)

matched_phrases = phrase_matcher(sentence)  

In the output, we will have all the ids of all the matched phrases along with their start and end indexes in the document as shown below:

[(5530044837203964789, 37, 39),
 (5530044837203964789, 402, 404),
 (5530044837203964789, 693, 694),
 (5530044837203964789, 1284, 1286),
 (5530044837203964789, 3059, 3061),
 (5530044837203964789, 3218, 3220),
 (5530044837203964789, 3753, 3754),
 (5530044837203964789, 5212, 5213),
 (5530044837203964789, 5287, 5288),
 (5530044837203964789, 6769, 6771),
 (5530044837203964789, 6781, 6783),
 (5530044837203964789, 7496, 7498),
 (5530044837203964789, 7635, 7637),
 (5530044837203964789, 8002, 8004),
 (5530044837203964789, 9461, 9462),
 (5530044837203964789, 9955, 9957),
 (5530044837203964789, 10784, 10785),
 (5530044837203964789, 11250, 11251),
 (5530044837203964789, 12290, 12291),
 (5530044837203964789, 12411, 12412),
 (5530044837203964789, 12455, 12456)]

To see the string value of the matched phrases, execute the following script:

for match_id, start, end in matched_phrases:  
    string_id = nlp.vocab.strings[match_id]  
    span = sentence[start:end]                   
    print(match_id, string_id, start, end, span.text)

In the output, you will see the strig value of the matched phrases as shown below:

5530044837203964789 AI 37 39 intelligent agents  
5530044837203964789 AI 402 404 machine learning  
5530044837203964789 AI 693 694 robots  
5530044837203964789 AI 1284 1286 machine learning  
5530044837203964789 AI 3059 3061 intelligent agents  
5530044837203964789 AI 3218 3220 machine learning  
5530044837203964789 AI 3753 3754 robots  
5530044837203964789 AI 5212 5213 robots  
5530044837203964789 AI 5287 5288 robots  
5530044837203964789 AI 6769 6771 machine learning  
5530044837203964789 AI 6781 6783 machine learning  
5530044837203964789 AI 7496 7498 machine learning  
5530044837203964789 AI 7635 7637 machine learning  
5530044837203964789 AI 8002 8004 machine learning  
5530044837203964789 AI 9461 9462 robots  
5530044837203964789 AI 9955 9957 machine learning  
5530044837203964789 AI 10784 10785 robots  
5530044837203964789 AI 11250 11251 robots  
5530044837203964789 AI 12290 12291 robots  
5530044837203964789 AI 12411 12412 robots  
5530044837203964789 AI 12455 12456 robots  

From the output, you can see all the three phrases that we tried to search along with their start and end index and the string ids.

Stop Words

Before we conclude this article, I just wanted to touch the concept of stop words. Stop words are English words such as "the", "a", "an" etc that do not have any meaning of their own. Stop words are often not very useful for NLP tasks such as text classification or language modeling. So it is often better to remove these stop words before further processing of the document.

The spaCy library contains 305 stop words. In addition, depending upon our requirements, we can also add or remove stop words from the spaCy library.

To see the default spaCy stop words, we can use stop_words attribute of the spaCy model as shown below:

import spacy  
sp = spacy.load('en_core_web_sm')  
print(sp.Defaults.stop_words)  

In the output, you will see all the sPacy stop words:

{'less', 'except', 'top', 'me', 'three', 'fifteen', 'a', 'is', 'those', 'all', 'then', 'everyone', 'without', 'must', 'has', 'any', 'anyhow', 'keep', 'through', 'bottom', 'get', 'indeed', 'it', 'still', 'ten', 'whatever', 'doing', 'though', 'eight', 'various', 'myself', 'across', 'wherever', 'himself', 'always', 'thus', 'am', 'after', 'should', 'perhaps', 'at', 'down', 'own', 'rather', 'regarding', 'which', 'anywhere', 'whence', 'would', 'been', 'how', 'herself', 'now', 'might', 'please', 'behind', 'every', 'seems', 'alone', 'from', 'via', 'its', 'become', 'hers', 'there', 'front', 'whose', 'before', 'against', 'whereafter', 'up', 'whither', 'two', 'five', 'eleven', 'why', 'below', 'out', 'whereas', 'serious', 'six', 'give', 'also', 'became', 'his', 'anyway', 'none', 'again', 'onto', 'else', 'have', 'few', 'thereby', 'whoever', 'yet', 'part', 'just', 'afterwards', 'mostly', 'see', 'hereby', 'not', 'can', 'once', 'therefore', 'together', 'whom', 'elsewhere', 'beforehand', 'themselves', 'with', 'seem', 'many', 'upon', 'former', 'are', 'who', 'becoming', 'formerly', 'between', 'cannot', 'him', 'that', 'first', 'more', 'although', 'whenever', 'under', 'whereby', 'my', 'whereupon', 'anyone', 'toward', 'by', 'four', 'since', 'amongst', 'move', 'each', 'forty', 'somehow', 'as', 'besides', 'used', 'if', 'name', 'when', 'ever', 'however', 'otherwise', 'hundred', 'moreover', 'your', 'sometimes', 'the', 'empty', 'another', 'where', 'her', 'enough', 'quite', 'throughout', 'anything', 'she', 'and', 'does', 'above', 'within', 'show', 'in', 'this', 'back', 'made', 'nobody', 'off', 're', 'meanwhile', 'than', 'neither', 'twenty', 'call', 'you', 'next', 'thereupon', 'therein', 'go', 'or', 'seemed', 'such', 'latterly', 'already', 'mine', 'yourself', 'an', 'amount', 'hereupon', 'namely', 'same', 'their', 'of', 'yours', 'could', 'be', 'done', 'whole', 'seeming', 'someone', 'these', 'towards', 'among', 'becomes', 'per', 'thru', 'beyond', 'beside', 'both', 'latter', 'ours', 'well', 'make', 'nowhere', 'about', 'were', 'others', 'due', 'yourselves', 'unless', 'thereafter', 'even', 'too', 'most', 'everything', 'our', 'something', 'did', 'using', 'full', 'while', 'will', 'only', 'nor', 'often', 'side', 'being', 'least', 'over', 'some', 'along', 'was', 'very', 'on', 'into', 'nine', 'noone', 'several', 'i', 'one', 'third', 'herein', 'but', 'further', 'here', 'whether', 'because', 'either', 'hereafter', 'really', 'so', 'somewhere', 'we', 'nevertheless', 'last', 'had', 'they', 'thence', 'almost', 'ca', 'everywhere', 'itself', 'no', 'ourselves', 'may', 'wherein', 'take', 'around', 'never', 'them', 'to', 'until', 'do', 'what', 'say', 'twelve', 'nothing', 'during', 'sixty', 'sometime', 'us', 'fifty', 'much', 'for', 'other', 'hence', 'he', 'put'}

You can also check if a word is a stop word or not. To do so, you can use the is_stop attribute as shown below:

sp.vocab['wonder'].is_stop  

Since "wonder" is not a spaCy stop word, you will see False in the output.

To add or remove stopwords in spaCy, you can use sp.Defaults.stop_words.add() and sp.Defaults.stop_words.remove() methods respectively.

sp.Defaults.stop_words.add('wonder')  

Next, we need to set the is_stop tag for wonder to 'True` as shown below:

sp.vocab['wonder'].is_stop = True  

Conclusion

Phrase and vocabulary matching is one of the most important natural language processing tasks. In this article, we continued our discussion about how to use Python to perform rule-based and phrase based matching. In addition, we also saw spaCy stop words. In the next article, we will see parts of speech tagging and named entity recognition in detail.

March 21, 2019 01:20 PM UTC

Talk Python to Me

#204 StaticFrame, like Pandas but safer

Remember back in math class when you would take a test? It wasn't enough to just write down the answer. What's the limit of this infinite summation? pi/2 Yes, but how did you get that number.

March 21, 2019 08:00 AM UTC

March 20, 2019

Mike Driscoll

Book Review: Mission Python

Book Formats

You can get this book as a paperback, PDF, Mobi, or ePub.

Book Contents

This book contains 12 chapters and 3 appendices across 280 pages. The code in this book is for Windows or Raspberry Pi. You will also need Python 3 for this book as it uses PyGame Zero, which requires Python 3.

Full Review

Mission Python attempts to teach you the Python programming language while also building a 2D video game using PyGame Zero. The introduction helpfully explains how to install Python and the dependencies needed for the game on Windows and Raspberry Pi. If you happen to be on Linux or Mac, you will have to figure out how to install it on your own.

The first chapter of the book instructs the reader on using Python’s IDLE application for developing your code. The rest of the chapter introduces you to writing the some of the the game. Basically you create the background for the game and learn about blitting. You also learn how to move a character in the game using your keyboard.

Chapter 2 digs into Python lists and how you can use them in conjunction with your game. The author uses list building and accessing elements within a list to help the reader create space related checklists and eventually a map of sorts. It’s a rather creative way to integrate Python lists and a video game at the same time.

The next step in the reader’s journey is learning about Python loops in chapter 3. Loops are used for printing the “map” to the screen. The map is basically a matrix of numbers. After describing how to do nested loops, the author uses that newfound knowledge to help the user create a room image using PyGame Zero’s drawing abilities.

For chapter 4, the author teaches the reader how to create the map and the automatic map maker too. Here you will learn how to use PyGame Zero to draw the map onscreen and debug the code when you inevitably run into issues.

Chapters 5 and 6 are about the space station’s equipment and focus on Python dictionaries. Here the reader will learn how to mix data types in Python, such as adding lists to dictionaries. You will also be learning how to load scenery into the rooms in your game.

The goal of chapter 7 is to teach you how to move your sprites about in the game. For this task, the reader learns how to hook up the keyboard to the game and the different versions of the sprites you will need to make movement look natural.

The rest of the book builds up various parts of the game itself. You will be learning how to use functions along the way and there are exercises spread throughout the book. Most of the rest of the book focuses on PyGame specific functions and methods and how to use them effectively.

This book seems well organized and pretty well written. I personally find the chapter titles a bit confusing as you can’t really tell from the titles alone what the chapter is actually about. The section names are sometimes written in an obtuse manner as well. However the content of the book is interesting and it’s fun to learn how to write a game in Python. I would give this book to someone who would like to learn all the moving parts of writing a game in Python. You won’t be learning game theory, but it’s still a good introduction to game development with nice, small chunks of code.

Mission Python by Sean McManus Amazon, No Starch

Other Book Reviews

• Serious Python: Black-Belt Advice on Deployment, Scalability, Testing, and More by Julien Danjou
• Python Testing with pytest by Brian Okken
• Module Programming with Python by Erik Westra
• Python Playground – Geeky Projects for the Curious Programmer by Mahesh Venkitachalam
• IPython Notebook Essentials by L. Felipe Martins
• Creating Apps in Kivy by Dusty Phillips

March 20, 2019 10:45 PM UTC

Continuum Analytics Blog

Announcing Public Anaconda Package Download Data

I’m very happy to announce that starting today, we will be publishing summarized download data for all conda packages served in the Anaconda Distribution, as well as the popular conda-forge and bioconda channels.  The dataset…

The post Announcing Public Anaconda Package Download Data appeared first on Anaconda.

March 20, 2019 06:24 PM UTC

Python Data

Comparing Machine Learning Methods

When working with data and modeling, its sometimes hard to determine what model you should use for a particular modeling project.  A quick way to find an algorithm that might work better than others is to run through an algorithm comparison loop to see how various models work against your data. In this post, I’ll be comparing machine learning methods using a few different sklearn algorithms.  As always, you can find a jupyter notebook for this article on my github here.

I’ve used Jason Brownlee’s article from 2016 as the basis for this article…I wanted to expand a bit on what he did as well as use a different dataset. In this article, we’ll be using the Indian Liver Disease dataset (found here).

From the dataset page:

This data set contains 416 liver patient records and 167 non liver patient records collected from North East of Andhra Pradesh, India. The “Dataset” column is a class label used to divide groups into liver patient (liver disease) or not (no disease). This data set contains 441 male patient records and 142 female patient records.

Let’s get started by setting up our imports that we’ll use.

import pandas as pd
import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"] = (20,10)

from sklearn import model_selection
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

Next, we’ll read in the data from the CSV file located in the local directory.

#read in the data
data = pd.read_csv('indian_liver_patient.csv')

If you do a head() of the dataframe, you’ll get a good feeling for the dataset.

We’ll use all columns except Gender for this tutorial. We could use gender by converting the gender to a numeric value (e.g., 0 for Male, 1 for Female) but for the purposes of this post, we’ll just skip this column.

data_to_use = data
del data_to_use['Gender']
data_to_use.dropna(inplace=True)

The ‘Dataset’ column is the value we are trying to predict…whether the user has liver disease or not so we’ll that as our “Y” and the other columns for our “X” array.

values = data_to_use.values

Y = values[:,9]
X = values[:,0:9]

Before we run our machine learning models, we need to set a random number to use to seed them. This can be any random number that you’d like it to be. Some people like to use a random number generator but for the purposes of this, I’ll just set it to 12 (it could just as easily be 1 or 3 or 1023 or any other number).

random_seed = 12

Now we need to set up our models that we’ll be testing out. We’ll set up a list of the models and give them each a name. Additionally, I’m going to set up the blank arrays/lists for the outcomes and the names of the models to use for comparison.

outcome = []
model_names = []
models = [('LogReg', LogisticRegression()), 
          ('SVM', SVC()), 
          ('DecTree', DecisionTreeClassifier()),
          ('KNN', KNeighborsClassifier()),
          ('LinDisc', LinearDiscriminantAnalysis()),
          ('GaussianNB', GaussianNB())]

We are going to use a k-fold validation to evaluate each algorithm and will run through each model with a for loop, running the analysis and then storing the outcomes into the lists we created above. We’ll use a 10-fold cross validation.

for model_name, model in models:
    k_fold_validation = model_selection.KFold(n_splits=10, random_state=random_seed)
    results = model_selection.cross_val_score(model, X, Y, cv=k_fold_validation, scoring='accuracy')
    outcome.append(results)
    model_names.append(model_name)
    output_message = "%s| Mean=%f STD=%f" % (model_name, results.mean(), results.std())
    print(output_message)

The output from this loop is:

LogReg| Mean=0.718633 STD=0.058744
SVM| Mean=0.715124 STD=0.058962
DecTree| Mean=0.637568 STD=0.108805
KNN| Mean=0.651301 STD=0.079872
LinDisc| Mean=0.716878 STD=0.050734
GaussianNB| Mean=0.554719 STD=0.081961

From the above, it looks like the Logistic Regression, Support Vector Machine and Linear Discrimination Analysis methods are providing the best results (based on the ‘mean’ values). Taking Jason’s lead, we can take a look at a box plot to see what the accuracy is for each cross validation fold, we can see just how good each does relative to each other and their means.

fig = plt.figure()
fig.suptitle('Machine Learning Model Comparison')
ax = fig.add_subplot(111)
plt.boxplot(outcome)
ax.set_xticklabels(model_names)
plt.show()

From the box plot, it is easy to see the three mentioned (Logistic Regression, Support Vector Machine and Linear Discrimination Analysis) are providing the better accuracies. From this outcome, we can then take this data and start working with these three models to see how we might be able to optimize the modeling process to see if one model works a bit better than others.

The post Comparing Machine Learning Methods appeared first on Python Data.

March 20, 2019 05:35 PM UTC

Anarcat

Securing registration email

I've been running my own email server basically forever. Recently, I've been thinking about possible attack vectors against my personal email. There's of course a lot of private information in that email address, and if someone manages to compromise my email account, they will see a lot of personal information. That's somewhat worrisome, but there are possibly more serious problems to worry about.

TL;DR: if you can, create a second email address to register on websites and use stronger protections on that account from your regular mail.

Hacking accounts through email

Strangely what keeps me up at night is more what kind of damage an attacker could do to other accounts I hold with that email address. Because basically every online service is backed by an email address, if someone controls my email address, they can do a password reset on every account I have online. In fact, some authentication systems just gave up on passwords algother and use the email system itself for authentication, essentially using the "password reset" feature as the authentication mechanism.

Some services have protections against this: for example, GitHub require a 2FA token when doing certain changes which the attacker hopefully wouldn't have (although phishing attacks have been getting better at bypassing those protections). Other services will warn you about the password change which might be useful, except the warning is usually sent... to the hacked email address, which doesn't help at all.

The solution: a separate mailbox

I had been using an extension (anarcat+register@example.com) to store registration mail in a separate folder for a while already. This allows me to bypass greylisting on the email address, for one. Greylisting is really annoying when you register on a service or do a password reset... The extension also allows me to sort those annoying emails in a separate folder automatically with a simple Sieve rule.

More recently, I have been forced to use a completely different email alias (register@example.com) on some services that dislike having plus signs (+) in email address, even though they are perfectly valid. That got me thinking about the security problem again: if I have a different alias why not make it a completely separate account and harden that against intrusion. With a separate account, I could enforce things like SSH-only access or 2FA that would be inconvenient for my main email address when I travel, because I sometimes log into webmail for example. Because I don't frequently need access to registration mail, it seemed like a good tradeoff.

So I created a second account, with a locked password and SSH-only authentication. That way the only way someone can compromise my "registration email" is by hacking my physical machine or the server directly, not by just bruteforcing a password.

Now of course I need to figure out which sites I'm registered on with a "non-registration" email (anarcat@example.com): before I thought of using the register@ alias, I sometimes used my normal address instead. So I'll have to track those down and reset those. But it seems I already blocked a large attack surface with a very simple change and that feels quite satisfying.

Implementation details

Using syncmaildir (SMD) to sync my email, the change was fairly simple. First I need to create a second SMD profile:

if [ $(hostname) = "marcos" ]; then
    exit 1
fi

SERVERNAME=smd-server-register
CLIENTNAME=$(hostname)-register
MAILBOX_LOCAL=Maildir/.register/
MAILBOX_REMOTE=Maildir
TRANSLATOR_LR="smd-translate -m move -d LR register"
TRANSLATOR_RL="smd-translate -m move -d RL register"
EXCLUDE="Maildir/.notmuch/hooks/* Maildir/.notmuch/xapian/*"

Very similar to the normal profile, except mails get stored in the already existing Maildir/.register/ and different SSH profile and translation rules are used. The new SSH profile is basically identical to the previous one:

# wrapper for smd
Host smd-server-register
    Hostname imap.anarc.at
    BatchMode yes
    Compression yes
    User register
    IdentitiesOnly yes
    IdentityFile ~/.ssh/id_ed25519_smd

Then we need to ignore the register folder in the normal configuration:

diff --git a/.smd/config.default b/.smd/config.default
index c42e3d0..74a8b54 100644
--- a/.smd/config.default
+++ b/.smd/config.default
@@ -59,7 +59,7 @@ TRANSLATOR_RL="smd-translate -m move -d RL default"
 # EXCLUDE_LOCAL="Mail/spam Mail/trash"
 # EXCLUDE_REMOTE="OtherMail/with%20spaces"
 #EXCLUDE="Maildir/.notmuch/hooks/* Maildir/.notmuch/xapian/*"
-EXCLUDE="Maildir/.notmuch/hooks/* Maildir/.notmuch/xapian/*"
+EXCLUDE="Maildir/.notmuch/hooks/* Maildir/.notmuch/xapian/* Maildir/.register/*"
 #EXCLUDE_LOCAL="$MAILBOX_LOCAL/.notmuch/hooks/* $MAILBOX_LOCAL/.notmuch/xapian/*"
 #EXCLUDE_REMOTE="$MAILBOX_REMOTE/.notmuch/hooks/* $MAILBOX_REMOTE/.notmuch/xapian/*"
 #EXCLUDE_REMOTE="Maildir/Koumbit Maildir/Koumbit* Maildir/Koumbit/* Maildir/Koumbit.INBOX.Archives/ Maildir/Koumbit.INBOX.Archives.2012/ Maildir/.notmuch/hooks/* Maildir/.notmuch/xapian/*"

And finally we add the new profile to the systemd services:

diff --git a/.config/systemd/user/smd-pull.service b/.config/systemd/user/smd-pull.service
index a841306..498391d 100644
--- a/.config/systemd/user/smd-pull.service
+++ b/.config/systemd/user/smd-pull.service
@@ -8,6 +8,7 @@ ConditionHost=!marcos
 Type=oneshot
 # --show-tags gives email counts
 ExecStart=/usr/bin/smd-pull --show-tags
+ExecStart=/usr/bin/smd-pull --show-tags register

 [Install]
 WantedBy=multi-user.target
diff --git a/.config/systemd/user/smd-push.service b/.config/systemd/user/smd-push.service
index 10d53c7..caa588e 100644
--- a/.config/systemd/user/smd-push.service
+++ b/.config/systemd/user/smd-push.service
@@ -8,6 +8,7 @@ ConditionHost=!marcos
 Type=oneshot
 # --show-tags gives email counts
 ExecStart=/usr/bin/smd-push --show-tags
+ExecStart=/usr/bin/smd-push --show-tags register

 [Install]
 WantedBy=multi-user.target

That's about it on the client side. On the server, the user is created with a locked password the mailbox moved over:

adduser --disabled-password register
mv ~anarcat/Maildir/.register/ ~register/Maildir/
chown -R register:register Maildir/

The SSH authentication key is added to .ssh/authorized_keys, and the alias is reversed:

--- a/aliases
+++ b/aliases
@@ -24,7 +24,7 @@ spamtrap: anarcat
 spampd: anarcat
 junk: anarcat
 devnull: /dev/null
-register: anarcat+register
+anarcat+register: register

 # various sandboxes
 anarcat-irc: anarcat

... and the email is also added to /etc/postgrey/whitelist_recipients.

That's it: I now have a hardened email service! Of course there are other ways to harden an email address. On-disk encryption comes to mind but that only works with password-based authentication from what I understand, which is something I want to avoid to remove bruteforce attacks.

Your advice and comments are of course very welcome, as usual

March 20, 2019 03:28 PM UTC

PyCharm

PyCharm 2019.1 RC 2

New in PyCharm 2019.1: completely redesigned Jupyter Notebooks, improved HTML & CSS quick documentation, custom themes, and more. Get the release candidate from our website

New in PyCharm

Custom Themes

We’ve now added official support for theme plugins, so be sure to check out the themes available in the Marketplace to see if there’s one you like. In addition to the popular third party Material Theme UI plugin, our own UI team has now published several plugins. For example, the Cyan Light theme:

If you don’t like any of the available UI themes, you can also make your own. Read more about creating UI themes on the IntelliJ blog.

Redesigned Database Connection Flow

PyCharm Professional Edition bundles the advanced database tools from JetBrains DataGrip, a tool that can manage databases from all major vendors. DataGrip has redesigned the flow for connecting to a database, and this is now also available in PyCharm.

If you’re working on a database-connected application, be sure to configure your database settings in PyCharm. With the database connected, you get schema-aware SQL completion in Python code, and you can quickly run a query embedded in Python code straight from the editor.

Further Improvements

• Various bugs in our completion for Angular were fixed. PyCharm Professional Edition bundles all HTML, CSS, and JavaScript features from JetBrains WebStorm.
• Read about all improvements in RC2 in the release notes

Interested?

Download the RC from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date.

If you’re on Ubuntu 16.04 or later, you can use snap to get PyCharm RC versions, and stay up to date. You can find the installation instructions on our website.

The release candidate (RC) is not an early access program (EAP) build, and does not bundle an EAP license. If you get PyCharm Professional Edition RC, you will either need a currently active PyCharm subscription, or you will receive a 30-day free trial.

March 20, 2019 02:57 PM UTC

Abhijeet Pal

How To Create A ‘Hello, World!’ Application With Django

Django is a high-level full stack open source web framework written in Python, that encourages rapid development and clean, pragmatic design. Django comes with lots of advance functionalities baked in which saves developers a lot of time. The simplicity Django offers lets developers focus more on writing the app instead of rewriting the same wheel. Since it’s release in 2003 Django has proven to be the most productive framework for Python developers to know more about Django read: Django – Web Framework For Perfectionists

In this article, we will create the traditional “Hello, World!” app, which will basically display the string ‘Hello, world!’ in the browser. This might be your first Django app so pay close attention to the core principles of Django which we will discuss later in the article.

Creating A Virtual Environment

Though this is an optional step yet it is highly recommended to use virtual environments in your projects, to know why are virtual environments so important read: How To A Create Virtual Environment for Python

In a nutshell, virtual environment lets you have an isolated space on your computer for different Python projects, ensuring that each of your projects can have its own set of dependencies and modules that won’t disrupt any of your other projects.

Let’s kick off by creating a virtual environment for our hello world project.

For Windows

cd Desktop
virtualenv django
cd django
Scripts\activate.bat

For Mac and Unix

mkdir django
cd django
python3 -m venv myenv
source django/bin/activate

Now you should see (django) prefixed in your terminal, which indicates that the virtual environment is successfully activated, if not then go through the guide again.

Installing Django In The Virtual Environment

Now, we need to Install Django on our virtual environment.

pip install Django

This will install the latest version of Django in our virtual Environment. To know more about Django installation read: How To Install Django

Creating A Django Project

Moving on, now time to build the Hello world project, this is a basic Django project. First, create a directory in your desktop named, hello_world and navigate into it.

cd Desktop
mkdir hello_world 
cd hello_world

Next, create a Django Project:

django-admin startproject hello_world_project

Executing this will invoke the django-admin.py script which will set up a new Django project instance name hello_world_project in the hello_world directory.

hello_world_project/
   manage.py
   hello_world_project/
      __init__.py
      settings.py
      urls.py
      wsgi.py

To know about these files and the project structure we highly recommend to read: Starting A Django Project

Nevertheless briefly going over the files,

manage.py – Command line utility lets you interact with your Django project.

__init__.py –  a blank Python script whose presence indicates to the Python interpreter that the directory is a Python package.

settings.py – Contains the configuration settings for the Django project.

urls.py – Contains URL patterns for the Django project.

wsgi.py – Contains WSGI configuration properties for the Django project.

Now, let’s apply migrations and test our project. Navigate into the Base directory ( i.e. the outer directory ) and run these commands.

cd hello_world_project
python manage.py migrate
python manage.py runserver

This will start the Django’s built-in server now open your preferred browser and navigate to this address http://127.0.0.1:8000/ if everything went well you should see the default Django’s welcome page.

Now press ctrl+c in the terminal window, to stop the server.

Creating A Django App

A Django project is a set of applications and configurations which combined make a full-fledged web application. Django apps are the sub-directories inside the Django project. The purpose of Django applications is to perform a particular task which in this case is to render ‘Hello, World!’.

Make sure you are at the outer directory where manage.py is and run the following.

 python manage.py startapp my_app

This will create another directory inside the project called my_app, now the project should look something like this.

├── db.sqlite3
├── hello_world_project
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   ├── wsgi.py
├── manage.py
└── my_app
    ├── __init__.py
    ├── admin.py
    ├── apps.py
    ├── migrations
    │   └── __init__.py
    ├── models.py
    ├── tests.py
    └── views.py

There are lots of new files in the apps which we haven’t seen before in the project,

admin.py – File with admin definitions for the app – such definitions are needed to access model class instances from the Django admin

apps.py – File with configuration parameters for the app.

models.py – File with database definitions (i.e., model classes) for the app.

tests.py – File with test definitions for the app.

views.py – File with view definitions (i.e., controller methods) for the app.

migrations – The directory that contains migrations applied to the app’s database definitions (i.e., model classes).

Now we need to add this app into the Installed apps list in Django’s settings so that Django can know about the app. Open your preferred text editor and open the settings.py file and scroll to the INSTALLED_APPS section. There you should see the list of built-in Django apps.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Add my_app below the preinstalled apps and save it. Note that user-defined apps should always be below the pre-installed apps because Django read the apps in the top to down fashion and our app may rely on the Django’s core apps.

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'my_app'
]

Creating Web App

Till now everything was about the configuration which is needed to be done for any web app, now it’s time to actually design the app. Suppose you were creating a hello world app without a framework you’d simply type Hello world into a text file, call it hello.html, and upload it to a directory on a web server somewhere.

Notice in this process you’ve specified two key pieces of information about that web page: its contents (the string Hello world) and its URL (for example, http://www.example.com/hello.html).

With Django, you specify those same two things, but in a different manner. The view function produces the contents of the page in the views.py file and the URL is specified in urls.py file.

Let’s create our first view, open views.py file of my_app and add the below lines.

from django.http import HttpResponse

def index(request):
    return HttpResponse('Hello, World!')

First, we imported the HttpResponse class from django.http module then we made a function that takes in a request and returns a HttpResponse object i.e. the string ‘Hello, World!’. Note that every view function must take atleast one parameter by convention called request.

In order to see this view in our browser, we need to map this view in our URL configurations. Open the urls.py file of the main project. Which should look like this.

from django.contrib import admin
from django.urls import path

urlpatterns = [
    path('admin/', admin.site.urls),
]

Now we need to tell Django explicitly that we need to activate the view for a particular URL

from django.contrib import admin
from django.urls import path
# imported views
from my_app import views

urlpatterns = [
    path('admin/', admin.site.urls),
    # configured the url
    path('',views.index, name="homepage")
]

First, we imported the views from my_app directory then in the URL patterns we added the path for the view which is the homepage hence blank string denoted with ' ' than we mapped this URL to our index view, and at last the optional argument name which we assign to homepage. This implies every request to the homepage should return the ‘Hello, world!’ string.

Now let’s test out our app save the files open terminal and run the development server.

python manage.py runserver

Now visit http://127.0.0.1:8000/ you should see Hello, World! written there.

Try adding random paths after the URL like http://127.0.0.1:8000/something this will return 404 pages because we haven’t mapped URLs for those pages.

In this article, we went through the creation of the traditional ‘Hello, world!’ app with Django if you have any question regarding it feel free to ask in the comment section below.

The post How To Create A ‘Hello, World!’ Application With Django appeared first on Django Central.

March 20, 2019 02:53 PM UTC

Trey Hunner

Unique sentinel values, identity checks, and when to use object() instead of None

Occasionally in Python (and in programming in general), you’ll need an object which can be uniquely identified. Sometimes this unique object represents a stop value or a skip value and sometimes it’s an initial value. But in each of these cases you want your object to stand out from the other objects you’re working with.

When you need a unique value (a sentinel value maybe) None is often the value to reach for. But sometimes None isn’t enough: sometimes None is ambiguous.

In this article we’ll talk about when None isn’t enough, I’ll show you how I create unique values when None doesn’t cut it, and we’ll see a few different uses for this technique.

Initial values and default values

Let’s re-implement a version of Python’s built-in min function.

1
2
3
4
5
6
7
8
9
10
11
12

def min(iterable, default=None):
    """Imperfect re-implementation of Python's built-in min function."""
    minimum = None
    for item in iterable:
        if minimum is None or item < minimum:
            minimum = item
    if minimum is not None:
        return minimum
    elif default is not None:
        return default
    else:
        raise ValueError("Empty iterable")

This min function, like the built-in one, returns the minimum value in the given iterable or raises an exception when an empty iterable is given unless a default value is specified (in which case the default is returned).

1
2
3
4
5
6
7
8
9
10
11
12
13

>>> min([4, 3, 8, 7])
3
>>> min([9])
9
>>> min([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 12, in min
ValueError: Empty iterable
>>> min([], default=9)
9
>>> min([4, 3, 8, 7], default=9)
3

This behavior is somewhat similar to the built-in min function, except our code is buggy!

There are two bugs here.

First, an iterable containing a single None value will be treated as if it was an empty iterable:

1
2
3
4
5
6
7

>>> min([None], default=0)
0
>>> min([None])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in min
ValueError: Empty iterable

Second, if we specify our default value as None this min function won’t accept it:

1
2
3
4
5
6
7

>>> min([], default='')
''
>>> min([], default=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 12, in min
ValueError: Empty iterable

Why is this happening?

It’s all about None.

Why is None a problem?

The first bug in our code is related to the initial value for minimum and the second is related to the default value for our default argument. In both cases, we’re using None to represent an unspecified or un-initialized value.

1
2
3
4
5
6
7
8
9
10
11
12

def min(iterable, default=None):
    """Imperfect re-implementation of Python's built-in min function."""
    minimum = None
    for item in iterable:
        if minimum is None or item < minimum:
            minimum = item
    if minimum is not None:
        return minimum
    elif default is not None:
        return default
    else:
        raise ValueError("Empty iterable")

Using None is a problem in both cases because None is both a valid value for default and a valid value in our iterable.

Python’s None value is useful for representing emptiness, but it isn’t magical, at least not any more magical than any other valid value.

If we need a truly unique value for our default state, we need to invent our own.

When None isn’t a valid input for your function, it’s perfectly fine to use it to represent a unique default or initial state. But None is often valid data, which means None is sometimes a poor choice for a unique initial state.

We’ll fix both of our bugs by using object(): a somewhat common convention for creating a truly unique value in Python.

First we’ll set minimum to a unique object:

1
2
3
4
5
6
7
8
9
10
11
12
13

def min(iterable, default=None):
    """Imperfect re-implementation of Python's built-in min function."""
    initial = object()
    minimum = initial
    for item in iterable:
        if minimum is initial or item < minimum:
            minimum = item
    if minimum is not initial:
        return minimum
    elif default is not None:
        return default
    else:
        raise ValueError("Empty iterable")

That initial variable holds our unique value so we can check for its presence later.

This fixes the first bug:

1
2
3
4
5
6
7

>>> min([None], default=0)
>>> min([None])
>>> min([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 13, in min
ValueError: Empty iterable

But not the second.

To fix the second bug we need to use a different default value for our default argument (other than None).

To do this, we’ll make a global “constant” (by convention) variable, INITIAL, outside our function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

INITIAL = object()


def min(iterable, default=INITIAL):
    """Imperfect re-implementation of Python's built-in min function."""
    minimum = INITIAL
    for item in iterable:
        if minimum is INITIAL or item < minimum:
            minimum = item
    if minimum is not INITIAL:
        return minimum
    elif default is not INITIAL:
        return default
    else:
        raise ValueError("Empty iterable")

Now our code works exactly how we’d hope it would:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

>>> min([None], default=0)
>>> min([None])
>>> min([], default=None)
>>> min([], default='')
''
>>> min([4, 3, 7, 8])
3
>>> min([4, 3, 7, 8], default=0)
3
>>> min([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 12, in min
ValueError: Empty iterable

That’s lovely… but what is this magical object() thing? Why does it work, how does it work, and when should we use it?

What is object()?

Every class in Python has a base class of object (in Python 3 that is… things were a bit weirder in Python 2).

So object is a class:

1
2
3
4

>>> object
<class 'object'>
>>> type(object)
<class 'type'>

When we call object we’re creating an “instance” of the object class, just as calling any other class (when given the correct arguments) will create instances of them:

1
2
3
4
5
6

>>> set()
set()
>>> bytearray()
bytearray(b'')
>>> frozenset()
frozenset()

So we’re creating an instance of object. But… why?

Well, an instance of object shouldn’t be seen as equal to any other object:

1
2
3
4
5
6
7
8
9
10

>>> x = object()
>>> y = object()
>>> x == y
False
>>> x == 4
False
>>> x == None
False
>>> x == []
False

Except itself:

1
2
3
4

>>> x = object()
>>> z = x
>>> x == z
True

Python’s None is similar, except that anyone can get access to this unique None object anywhere in their code by just typing None.

1
2
3
4
5
6
7
8

>>> x = None
>>> y = None
>>> x == y
True
>>> x = object()
>>> y = object()
>>> x == y
False

We needed a placeholder value in our code. None is a lovely placeholder as long as we don’t need to worry about distinguishing between our None and their None.

If None is valid data, it’s no longer just a placeholder. At that point, we need to start reaching for object() instead.

Equality vs identity

I noted that object() isn’t equal to anything else. But we weren’t actually checking for equality (using == or !=) in our function:

Instead of == and !=, we used is and is not.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

INITIAL = object()


def min(iterable, default=INITIAL):
    """Imperfect re-implementation of Python's built-in min function."""
    minimum = INITIAL
    for item in iterable:
        if minimum is INITIAL or item < minimum:
            minimum = item
    if minimum is not INITIAL:
        return minimum
    elif default is not INITIAL:
        return default
    else:
        raise ValueError("Empty iterable")

While == and != are equality operators, is and is not are identity operators.

Python’s is operator asks about the identity of an object: are the two objects on either side of the is operator actually the same exact object.

We’re not just asking are they equal, but are they stored in the same place in memory and in fact refer to the same exact object.

Two of the variables below (x and z) point to the same object:

1
2
3

>>> x = object()
>>> y = object()
>>> z = x

So while y has a unique ID in memory, x and z do not:

1
2
3
4
5
6

>>> id(x)
140079600030400
>>> id(y)
140079561403808
>>> id(z)
140079600030400

Which means x is identical to z:

1
2
3
4

>>> x is y
False
>>> x is z
True

By default, Python’s == operator delegates to is. Meaning unless two variables point to the exact some object in memory, == will return False:

1
2
3
4
5
6
7
8
9

>>> x = object()
>>> y = object()
>>> z = x
>>> x == x
True
>>> x == y
False
>>> x == z
True

This is true by default… but many objects in Python overload the == operator to do much more useful things when we ask about equality.

1
2
3
4
5
6
7
8

>>> 0 == 0.0
True
>>> [1, 2, 3] == [1, 2, 3]
True
>>> (1, 2) == (1, 3)
False
>>> {} == {}
True

Each object can customize the behavior of == to answer whatever question they’d like.

Which means someone could make a class like this:

1
2
3
4

>>> class AlwaysEqual:
...     def __eq__(self, other):
...         return True
...

And suddenly our assumption about == with object() (or any other value) will fail us:

1
2
3
4
5
6

>>> x = object()
>>> y = AlwaysEqual()
>>> x is y
False
>>> x == y
True

Use identity to compare unique objects

The is operator, unlike ==, is not overloadable. Unlike with ==, there’s no way to control or change what happens when you say x is y.

There’s a __eq__ method, but there’s no such thing as a __is__ method. Which means the is operator will never lie to you: it will always tell you whether two objects are one in the same.

If we use is instead of ==, we could actually use any unique object to represent our unique INITIAL value.

Even an empty list:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

INITIAL = []


def min(iterable, default=INITIAL):
    """Imperfect re-implementation of Python's built-in min function."""
    minimum = INITIAL
    for item in iterable:
        if minimum is INITIAL or item < minimum:
            minimum = item
    if minimum is not INITIAL:
        return minimum
    elif default is not INITIAL:
        return default
    else:
        raise ValueError("Empty iterable")

An empty list might seem problematic in the same way as None was: but they’re actually quite different.

We don’t have any of the same issues as we did with None before:

1
2
3
4
5
6

>>> min([[]], default=0)
[]
>>> min([[]])
[]
>>> min([], default=[])
[]

The reason is that None is a singleton value. That means that whenever you say None in your Python code, you’re referencing the exact same None object every time.

1
2
3
4
5
6

>>> x = None
>>> y = None
>>> x is y
True
>>> id(x), id(y)
(94548887510464, 94548887510464)

Whereas every empty list we make creates a brand new list object:

1
2
3
4
5
6

>>> x = []
>>> y = []
>>> x is y
False
>>> id(x), id(y)
(140079561624776, 140079598927432)

So while two independent empty lists may be equal, they aren’t the same object:

1
2
3
4
5
6

>>> x = []
>>> y = []
>>> x == y
True
>>> x is y
False

The objects that those x and y variables point to have the same value but are not actually the same object.

None is a placeholder value

Python’s None is lovely. None is a universal placeholder value. Need a placeholder? Great! Python has a great placeholder value and it’s called None!

There are lots of places where Python itself actually uses None as a placeholder value also.

If you pass no arguments to the string split method, that’s the same as passing a separator value of None:

1
2
3
4
5

>>> s = "hello world"
>>> s.split()
['hello', 'world']
>>> s.split(None)
['hello', 'world']

If you pass in a key function of None to the sorted builtin, that’s the same as passing in no key function at all:

1
2
3
4

>>> sorted(s, key=None)
[' ', 'd', 'e', 'h', 'l', 'l', 'l', 'o', 'o', 'r', 'w']
>>> sorted(s)
[' ', 'd', 'e', 'h', 'l', 'l', 'l', 'o', 'o', 'r', 'w']

Python loves using None as a placeholder because it’s often a pretty great placeholder value.

The issue with None only appears if someone else could reasonably be using None as a non-placeholder input to our function. This is often the case when the caller of a function has a placeholder values (often None) in their inputs and the author of that function (that’s us) needs a separate unique placeholder.

Using None to represent two different things at once is like having two identical-looking bookmarks in the same book: it’s confusing!

Creating unique non-None placeholders: why object()?

When we made that INITIAL value before, we were sort of inventing our own None-like object: an object that we could uniquely reference by using the is operator.

That INITIAL object we made should be completely unique: it shouldn’t ever be seen in any arbitrary input that may be given to our function (unless someone made the strange decision to import INITIAL and reference it specifically).

Why object() though? After all we could have used any unique object by creating an instance of pretty much any class:

1
2
3
4
5

>>> INITIAL = []
>>> INITIAL == []
True
>>> INITIAL is []
False

Though it might have been even more clear to create our own class just for this purpose:

1
2
3
4

class DummyClass:
    """Class that just creates unique objects."""

INITIAL = DummyClass()

But I’d argue that object() is the “right” thing to use here.

Everyone knows what [] means, but object() is mysterious, which is actually the reason I think it’s a good choice in this case.

When we see an empty list we expect that list to be used as a list and when we see a class instance, we expect that class to do something. But we don’t actually want this object to do anything: we only care about the uniqueness of this new object.

We could have done this:

1

>>> INITIAL = ['completely unique value']

But I find using object() less confusing than this because it’s clear: readers won’t have a chance to be confused by the listy-ness of a list.

1

>>> INITIAL = object()  # completely unique value

Also if a confused developer Googles “what is object() in Python?” they might end up with some sort of explanation.

Other cases for non-None placeholders

There’s a word I’ve been avoiding using up to this point. I’ve only been avoiding it because I think I typically misuse it (or rather overuse it). The word is sentinel value.

I suspect I overuse this word because I use it to mean any unique placeholder value, such as the INITIAL object we made before. But most definitions I’ve seen use “sentinel value” to specifically mean a value which indicates the end of a list, a loop, or an algorithm.

Sentinel values are a thing that, when seen, indicate that something has finished. I think of this as a stop value: when you see a sentinel value it’s a signal that the loop or algorithm that you’re in should terminate.

Before we weren’t using a stop value so much as an initial value.

Here’s an example of a stop value; a true sentinel value:

1
2
3
4
5
6
7
8
9
10

from itertools import zip_longest

SENTINEL = object()

def strict_zip(*iterables):
    """Variation of ``zip`` which requires equal-length iterables."""
    for values in zip_longest(*iterables, fillvalue=SENTINEL):
        if SENTINEL in values:
            raise ValueError("Given iterables must have the same length.")
        yield values

We’re using the unique SENTINEL value above to signal that we need to stop looping and raise an exception. The presence of this value indicates that one of our iterables was a different length than the others and we need to handle this error case.

Rely on identity checks for unique values

Note that we’re implicitly relying on == above because we’re saying if SENTINEL in values which actually loops over values looking for a value that is equal to SENTINEL.

If we wanted to be more strict (and possibly more efficient) we could rely on is, but we’d need to do some looping ourselves. Fortunately Python’s any function and a generator expression would make that a bit easier:

1
2
3
4
5
6
7
8
9
10

from itertools import zip_longest

SENTINEL = object()

def strict_zip(*iterables):
    """Variation of ``zip`` which requires equal-length iterables."""
    for values in zip_longest(*iterables, fillvalue=SENTINEL):
        if any(v is SENTINEL for v in values):
            raise ValueError("Given iterables must have the same length.")
        yield values

I’m fine with either of these functions. The first is a bit more readable even though this one is arguably a bit more correct.

Identity checks are often faster than equality checks (== has to call the __eq__ method, but is does a straight memory ID check). But identity checks are also a bit more correct: if it’s uniqueness we care about, a unique memory location is the ultimate uniqueness check.

When writing code that uses a unique object, it’s wise to rely on identity rather than equality if you can.

This is what is was made for

If we care about equality (the value of an object) we use ==, if we care about identity (the memory location) we use is.

If you search my Python code for is you’ll pretty much only find the following things:

1. x is None (this is the most common thing you’ll see)
2. x is True or x is False (sometimes my tests get picky about True vs truthiness)
3. iter(x) is x (iterators are a different Python rabbit hole)
4. x is some_unique_object

Those first two are checking for a singleton value (as recommended by PEP 8). The third one is checking if we’ve seen the same object twice (an iterator in this case). And the fourth one is checking for the presence of these unique values we’ve been discussing.

The is operator checks whether two objects are exactly the same object in memory. You never want to use the is operator except for true identity checks: singletons (like None, True, and False), checking for the same object again, and checking for our own unique values (sentinels, as I usually call them).

So when would we use object()?

Oftentimes None is both the easy answer and the right answer for a unique placeholder value in Python, but sometimes you just need to invent your own unique placeholder value. In those cases object() is a great tool to have in your Python toolbox.

When would we actually use object() for a uniqueness check in our own code?

I can think of a few cases:

1. Unique initial values: a starting value that should be distinguished from values seen later (default and initial in our min function)
2. Unique stop values: a value whose presence tells us to stop looping/processing (a true sentinel value, as in strict_zip)
3. Unique skip values: a value whose presence should be treated as an empty value to be skipped over (we didn’t see this, but it comes up with utilities like itertools.zip_longest sometimes)

I hope this meandering through unique values has given you something (some non-None things) to think about.

May your None values be unambiguous and your identity checks be truly unique.

March 20, 2019 02:30 PM UTC

Real Python

13 Project Ideas for Intermediate Python Developers

Learning the basics of Python is a wonderful experience. But the euphoria of just learning can be replaced by the hunger for hands-on projects. It’s normal to want to build projects, hence the need for project ideas.

The problem though is that some projects are either too simple for an intermediate Python developer or too hard. This article will suggest projects you can work on as an intermediate Python developer. These project ideas will provide the appropriate level of challenge for you.

In this article, you’ll learn:

• The importance of building projects
• The major platforms you can build projects for
• Thirteen project ideas you can work on
• Some tips for working on projects

The Importance of Building Projects

Working on projects is vital to pushing your career as a Python developer forward. They make you apply the skills and knowledge you’re acquiring.

Projects can help you:

• Build confidence: You will believe more in your ability to create software regardless the level of complexity.

• Explore other technologies: You will learn about other technologies needed in building a complete product such as databases, servers, and other languages.

• Understand programming concepts better: You will learn to write code better and understand concepts such as design patterns and object-oriented programming.

• Experience a complete software development life cycle: You will learn how to plan before writing code, manage the coding process and update software.

There is a lot to gain from building projects as a Python developer.

Choosing a Project Platform

You need to build your software to run on a platform so that people who lack certain technical knowledge can use your software. The web, desktop, and command-line are the three major platforms you’ll want to build your projects for.

Web

Web applications are applications that run on the web, they can be accessed on any device without being downloaded, provided there is access to the internet. If you want your projects to be accessible by everyone with internet access, it needs to be a web application.

A web application has a back end and front end. The back end is the part where the business logic is: your back-end code will manipulate and store data. The front end is the interface of the application: your front-end code will determine the look of a web application.

As an intermediate Python developer, your major focus will be the back-end code. However, the front-end code is important too, so you will need some knowledge of HTML, CSS, and maybe JavaScript to create a simple-looking interface. Just the basics will be enough.

Another option is to use Python for both the front end and back end. Thanks to the anvil library, which eliminates the need for HTML, CSS, and JavaScript, you can focus on Python code alone.

You can build web applications with Python through web frameworks such as django and flask. The list of frameworks for building web applications using Python is long. There are plenty to choose from, but django and flask remain the most popular web frameworks.

Desktop GUI

Every time you perform a task on your PC, be it a desktop or laptop, it is through an application. As an intermediate Python developer, you can make your own desktop applications.

You do not have to learn any front-end technology to create your own Graphical User Interface (GUI) applications, as you saw with web applications. You can build all the parts using Python.

There are frameworks for building your desktop applications. PySimpleGUI is one of them, and it’s pretty user-friendly for an intermediate Python developer.

An advanced GUI framework like PyQt5 is quite powerful, but it may have a steep learning curve.

The software you create for the Desktop GUI is able to work on any of the Windows, Linux, or Mac operating systems. All you have to do after creating the project is compile it to an executable for your operating system of choice.

Command-Line

Command-line applications are those applications that work in a console window. This is the command prompt on Windows and the Terminal on Linux and Mac.

You’d click to use a web or GUI application, but you’d type in commands for command-line applications. Users of command-line applications need to have some technical knowledge since they’ll need to use commands.

Command-line applications may not be as beautiful or easy to use as web or GUI applications, but that doesn’t make them less powerful than web or GUI applications.

You can improve the look of your command-line applications by applying colors to the text. There are libraries you can use for coloring, such as colorama and colored. You can spice things up and use some color.

You can use frameworks such as docopt, argparse, and click to build your applications.

Web Project Ideas

In this section, you’ll see project ideas for the web. These project ideas can be classified as utility and education tools.

Here are the project ideas:

• Content Aggregator
• Regex Query Tool
• URL Shortener
• Post-It Note
• Quiz Application

Content Aggregator

Content is king. It exists everywhere on the web, from blogs to social media platforms. To keep up, you need to search for new information on the internet constantly. One way to stay updated is to check all the sites manually to see what the new posts are. But this is time consuming and quite tiring.

This is where the content aggregator comes in: A content aggregator fetches information from various places online and gathers all of that information in one place. Therefore, you don’t have to visit multiple sites to get the latest info: one website is enough.

With the content aggregator, all of the latest information can be gotten from one site that aggregates all the content. People can see the posts that interest them and can decide to find out more about them without traipsing all over the internet.

Examples of Content Aggregators

Here are some implementations of the Content Aggregator idea:

• AllTop
• Hvper

Technical Details

The main objective of this project idea is to aggregate content. First, you need to know what sites you’ll want the Content Aggregator to get content from. Then, you can use libraries such as requests for sending HTTP requests and BeautifulSoup to parse and scrape the necessary content from the sites.

Your application can implement its content aggregation as a background process. Libraries such as celery or apscheduler can help with that. You can try out apscheduler. It’s great for small background processes.

After scraping content from various sites, you’ll need to save it somewhere. So, you’ll use a database to save the scraped content.

Extra Challenge

For a tougher challenge, you can add more websites. This will help you learn how to study and extract information from websites.

You can also have users subscribe to certain sites that you aggregate. Then, at the end of the day, the content aggregator will send the articles for that day to the email address of the user.

Regex Query Tool

You and I deal with text daily. This article, which is also text, has a structure. This makes it easier for you to understand. Sometimes, you need to find certain information in text, and using the regular search tool in text editors can be ineffective.

This is where the Regex Query Tool comes in. A regex is a set of strings, so the regex query tool will check for the validity of the queries. When the regex matches patterns in the text, it tells the user and highlights the matched patterns. So, your Regex Query Tool will check the validity of the regex strings passed in by the user.

With the Regex Query Tool, users can quickly check the validity of their regex strings on the web. This makes it easier for them, instead of having to check the strings with a text editor.

Examples of Regex Query Tools

Here are some implementations of the Regex Query Tool idea:

• FreeFormatter
• RegexTester

Technical Details

The main objective of this type of project is to tell the user the validity of the inputted query strings. You can make it give a positive or negative response such as Query String Is Valid and Query String Is Invalid, implementing the positive response in green and the negative in red.

You don’t have to implement the query tool from scratch. You can use Python’s standard re library, which you can use to run the query strings on the inputted text. The re library will return None when the query string matches nothing, and it’ll return the matched strings when positive.

Some users may not understand regex fully, so you can make a page to explain how regex works. You can make documentation that is interesting enough to keep the users excited about learning and understanding regex.

Extra Challenge

Making a project that just returns the validity of the regex is fine. But you can also add a replacement feature. This means the application will check for the validity of the regex and also allow users to replace the matched strings with something else. So the tool is no longer a find tool but also a replace tool.

URL Shortener

URLs can be extremely long and not user-friendly. When people share links or even try to remember a URL, it’s difficult because most URLs are filled with more difficult characters and don’t form meaningful words.

This is where the URL Shortener comes in. A URL Shortener reduces the characters or letters in a URL, making them easier to read and remember. A URL like xyz.com/wwryb78&svnhkn%sghq?sfiyh can be shortened to xyz.com/piojwr.

With the URL Shortener, URLs become a joy to work with.

Examples of URL Shorteners

Here are some implementations of the URL Shortener idea:

• Bitly
• MeShort

Technical Details

The main objective of this project idea is to shorten URLs. The main task the application will accomplish is to shorten URLs and then redirect users to the original URL when the shortened URL is visited.

In the application, the users will input the original URL, and they will get the new, shortened URL as the result. To do this, you can use a combination of the random and string modules to generate the characters for the shortened URL.

Since users will visit the shortened URL days, months, or even years after, you’ll need to save the original and shortened URLs in a database. When a request comes in, the application checks if the URL exists and redirects to the original, or else it redirects to a 404 page.

Extra Challenge

Generating a shortened URL with random characters makes for a better URL than the long, random ones. But, you can make the result better for the users. You can add a feature to customize URLs, so the users can customize the generated URLs themselves.

Without a doubt, a custom xyz.com/mysite URL is better than a randomly generated xyz.com/piojwr URL.

Post-It Note

It’s human to have many thoughts and ideas in a day, but it’s also human to forget. One way to work around forgetting things is to jot them down before they disappear into thin air. While some of forgotten thoughts and ideas may be trivial, some can be quite powerful.

This is where a Post-It note comes in: A Post-It note is a small paper with low-tack adhesive at the back, making it attachable to surfaces such as documents, walls. Post-It notes make it easier to jot things down. The Post-It note project idea is something similar. It allows users to jot things down, making them accessible anywhere, since it’s a web application.

With the Post-It note, people can now jot things down anywhere, without the fear of forgetting things or misplacing the notes—which is a possibility with physical notes.

Examples of Post-It Notes

Here are some implementations of the Post-It Note idea:

• Pinup
• Note.ly

Technical Details

The main objective of this project is to allow users to jot down thoughts. This means that each user will have their own notes, so the application will need to have an account creation feature. This ensures that the notes of each user remain private to them.

django comes with a user authentication system, so it may be a good choice. You can use other frameworks like bottle or flask, but you’ll have to implement the user authentication system on your own.

Since users may need to separate their notes under different sections, implementing a feature to allow users to categorize their notes will make the application more useful.

As an example, you may need to have notes on algorithms and data structures, so you’ll need to be able to separate the notes in those categories.

You’ll need to store the information and notes of each user, so a database becomes an essential part of this project. The MySQLdb module can be used if you want to use a MySQL database or the psycopg2 module for a PostgreSQL database. There are other modules you can use, but it all depends on the database you choose to use.

Extra Challenge

Since it’s human for users to forget their ideas, it’s also human for them to forget that they even made a note somewhere. You can add a feature to remind users of their notes. This feature will allow users to set a time for the reminder, so the application will send the reminder to the users when it’s time, by email.

Quiz Application

Knowledge is power. There are so many things in the world to learn, and quizzes help in testing the understanding of those concepts. You, as an intermediate Python developer, do not have to understand everything about the language. Taking tests is one way to find out things you don’t fully understand.

This is where the Quiz Application comes in. The Quiz Application will present questions to the users and expect the right answers to those questions. Think of the Quiz Application as a kind of questionnaire.

With the Quiz Application, special users you can call administrators will be allowed to create tests, so regular users can answer the questions and test their understanding of the topics in the quiz.

Examples of Quiz Applications

Here are some implementations of the Quiz Application idea:

• myQuiz
• Kahoot

Technical Details

The main objective of this project is to set quizzes and have people answer them. Therefore, users should be able to set questions, and other users should be able to answer those questions. The application will then display the final score and the right answers.

If you want users to be able to have a record of their scores, you may have to implement an account creation feature.

Users creating the tests should be able to create tests with the questions and answers by simply uploading a text file. The text file will have a format that you can decide, so the application can convert from a file to a quiz.

You’ll need to implement a database for this project. The database will store the questions, possible answers, correct answers, and the scores for each user.

Extra Challenge

For more of a challenge, you can allow users to add timers to the quizzes. This way, the creators of a quiz can determine how many seconds or minutes a user should spend on each question in the quiz.

It would be great to also have a quiz-sharing feature, so users can share interesting quizzes with their friends on other platforms.

GUI Project Ideas

In this section, you’ll see project ideas for Graphical User Interfaces. These project ideas can be classified as entertainment, finance, and utility tools.

Here’s are the project ideas:

• MP3 Player
• Alarm Tool
• File Manager
• Expense Tracker

MP3 Player

Audio is as important as text today if not more important. Since audio files are digital files, you’ll need a tool that can play them. Without a player, you’ll never be able to listen to the contents of an audio file.

This is where the MP3 Player comes in. The MP3 Player is a device for playing MP3s and other digital audio files. This MP3 Player GUI project idea attempts to emulate the physical MP3 Player. You can build software that allows you play an MP3 files on your desktop or laptop computer.

When you are done building the MP3 Player project, users can play their MP3 files and other digital audio files without having to purchase a physical MP3 Player. They’ll be able to play the MP3 files using their computers.

Examples of MP3 Players

Here are some implementations of the MP3 Player idea:

• MusicBee
• Foobar2000

Technical Details

The main objective of this project is to allow users to play MP3 and digital audio files. To be engaging for users, the application has to have a simple but beautiful user interface.

You can have an interface for listing the available MP3 files. You can also give users the option to list other digital audio files that are not MP3.

The users will also expect the MP3 Player to have an interface that shows information on the file that is playing. Some of the information you can include are the name of the file, its length, the amount played, and the amount not played, in minutes and seconds.

Python has libraries that can play audio files, such as pygame, which allows you to work with multimedia files in few lines of code. You can also check out pymedia and simpleaudio.

These libraries can handle a lot of digital audio files. They can handle other file types, not just the MP3 files.

You can also implement a feature that allows users to create a playlist. To do this, you’ll need a database to store information on the created playlists. Python’s sqlite3 module allows you to use the SQLite database.

The SQLite database is a better option in this case, because it is file based and easier to set up than other SQL databases. While SQLite is file based, it is better for saving data than a regular file.

Extra Challenge

For a more exciting challenge, you can add a feature to allow the MP3 player to repeat currently playing files or even shuffle the list of files to be played.

It’s also possible to implement a feature that allows users to increase and decrease the playing speed of the audio file. Users will find this interesting, as they’ll be able to play files at a slower or faster pace than usual.

Alarm Tool

As they say, “Time and tide wait for no man.” But with a lot of things going on in our lives, it’s difficult to not lose track of time. To be able to keep track of time, a reminder is needed.

This is where the Alarm Tool comes in. An alarm is a device that gives an audio or visual signal about a certain condition. This Alarm Tool project idea is an attempt to build an alarm as software. The Alarm Tool gives an audio signal when a certain condition is met. The set time is the certain condition in this case.

With the Alarm Tool, users can set alarms to remind them of things at certain times of the day. The Alarm Tool project will work from the user’s laptop or desktop device, so they do not have to purchase a physical timer.

Examples of Alarm Tools

Here are some implementations of the Alarm Tool idea:

• FreeAlarmClock
• TimerForMac

Technical Details

The main objective of this project is to activate audio signals at certain times of the day. So, timing and the audio signal to be played are the most important parts of the Alarm Tool.

The Alarm Tool should allow users to create, edit, and delete alarms. It should also have an interface that lists all the alarms, provided they have not being deleted by the user. So, it should list the active and inactive alarms.

Since it is an alarm, the application has to play tones at the set time. There are libraries for playing audio, like the pygame library.

In your code logic, the application has to keep checking for set alarm times. When the time is reached, it triggers a function to play the alarm tone.

Since the application will check for set alarm times, it means the application has to save the alarms in a database. The database should store things like the alarm date, time, and tone location.

Extra Challenge

As an extra feature, you can allow users to set recurring alarms. They’ll be able to set alarms that will ring at a certain time on certain days of the week, every week. As an example, an alarm can be set at 2:00 PM every Monday.

You can also add a snooze feature, so your users can snooze alarms instead of only dismissing them.

File Manager

The number of files on the personal computer of an average PC user is pretty high. If all of those files were placed in a single directory, it would be difficult to navigate and find files or directories. So, there is a need to arrange the files and manage them properly.

This is where a file manager comes in. A file manager allows users to manage files and directories through a user interface. While files can be managed through the command-line, not all users know how to do that.

With a file manager, users can arrange, access, and administer their files and directories properly without knowing how to use the command line. Some of the tasks a file manager allows users to perform includes copying, moving, and renaming files or directories.

Examples of File Manager Tools

Here are some implementations of the File Manager idea:

• FreeCommander
• Explorer++

Technical Details

The main objective of the file manager project is to give users an interface to manage their files. Users want a file manager that has a file management tool that looks good and is easy to use.

You can use the PySimpleGUI library to create unique user interfaces with a powerful widget, without having to deal with a lot of complexity.

Your users should be able to perform simple tasks like creating new directories or empty text files. They should also be able to copy and move files or directories.

The sys, os, and shutil libraries will be quite useful for this project, as they can be used to execute actions on the files in the background, while the user clicks away.

The grid and list views are popular views today, so you can implement both in the application. This gives the user the option to choose which view option is suitable for them.

Extra Challenge

To make the file manager a bit more advanced, you can implement a search feature. So users can search for files and directories without having to find them manually.

You can also implement a sort feature. This will allow users to sort files according to different orders, such as time, alphabetical order, or size.

Expense Tracker

We have daily expenses, from groceries to clothing to bills. There are so many expenses that it’s normal to lose track of them and keep spending till we’re almost out of cash. A tracker can help people watch their expenses.

This is where the expense tracker comes in. An expense tracker is a software tool that allows users to keep track of their expenses. It can also analyze the expenses, depending on how advanced it is, but let’s keep it simple for now.

With the expense tracker, users can set a budget and track their spending so as to make better financial decisions.

Examples of Expense Trackers

Here are some implementations of the Expense Tracker idea:

• Buddi
• GnuCash

Technical Details

The main objective of this project is to keep track of the user’s expenses. Some statistical analysis has to be done to be able to give users correct information on their expenses and help them spend better.

While tracking the expenses is the key thing, a good interface is also important. With PySimpleGUI, you can create a unique interface to improve the experience of the users.

PyData libraries such as pandas and matplotlib can be helpful for building the expense tracker.

The pandas library can be used for the data analysis, and the matplotlib library can be used for plotting graphs. Graphs will give the users a visual representation of their expenses, and a visual representation is usually easier to understand.

The application will receive data from the users. The data here is the inputted expenses. So, you’ll have to store the expenses in a database. The SQLite database is a good database choice for this project since it can be set up quickly. You can use sqlite3 module for the SQLite database.

Extra Challenge

For your users to benefit from this project, they’ll have to input their expenses regularly, which might slip their mind. It could be useful for you to implement a reminder feature. So the application will send a notification at certain times of the day or the week, reminding them to make use of the expense tracker.

Command-Line Project Ideas

In this section, you’ll see project ideas for the command-line. The project ideas discussed can be classified as utility tools.

Here’s are the project ideas:

• Contact Book
• File Connectivity Checker
• Bulk File Rename Tool
• Directory Tree Generator

Contact Book

We come across lots of people daily. We make acquaintances and friends. We get their contacts to keep in touch later on. Sadly, keeping the received contact details can be hard. One way to do this is to write the contact details down. But this is not secure as the physical book can easily be lost.

This is where the Contact Book project comes in. A contact book is a tool for saving a contact’s details, such as name, address, phone number, and email address. With this contact book project, you can build a software tool that people can use to save and find contact details.

With the contact book project idea, users can save their contacts with less risk of losing the saved contact details. It’ll always be accessible from their computer, through the command-line.

Examples of Contact Book Tools

There are Contact Book applications, but it’s rare to find command-line Contact Book products, as most are web, mobile, or GUI applications.

Here are some implementations of the Contact Book idea:

• Simple Contacts
• Pobuca Connect

Technical Details

The main objective of this project is to save contact details. It’s important that you set up the commands users can use to enter the contact details. You can use the argparse or click command-line frameworks. They abstract a lot of complex stuff, so you only have to focus on the logic to be run when executing commands.

Some features you should implement include the commands to delete contacts, update contact information, and list saved contacts. You can also allow users to list contacts using different parameters, such as alphabetical order or contact creation date.

Since it’s a command-line project, the SQLite database will be fine for saving contacts. SQLite is user-friendly to set up. You may save the contact details in a file, but a file will not offer the benefits you can gain from using SQLite, such as performance and security.

To use the SQLite database in this project, the Python sqlite3 module will be very useful.

Extra Challenge

Remember how the database is stored on the user’s computer? What if something happens, like the user losing their files? It means they’ll also lose the contact details.

You can challenge yourself further and backup the database to an online storage platform. To do this, you can upload the database files to the cloud at certain intervals.

You can also add a command that allows users to backup the database themselves. This way, the user can still have access to the contacts if the database file is lost.

You should note that you may need some form of identification, so the contact book can tell which database file belongs to which user. Implementing a user authentication feature is one way to go about it.

Site Connectivity Checker

When you visit a URL, you expect to get the requested pages on your browser. But this is not always the case. Sometimes, sites can be down, so you won’t get the desired results. Instead, you’ll be presented with error messages. You can keep trying a site that is down, till it comes up and you get the information you need.

This is where the Site Connectivity Checker project comes in. The Site Connectivity Checker visits a URL and returns the status of the URL: it is either live or not. The Site Connectivity Checker will visit the URL at intervals, returning the results of each visit.

Instead of manually visiting a URL, a Site Connectivity Checker can do all of that manual work for you. This way, you’ll only get the results of the check without having to spend time on the browser, waiting for the site to go live.

Examples of Site Connectivity Checkers

Here are some implementations of the Site Connectivity Checker idea:

• Ping
• Site24x7

Technical Details

The main objective of this project is to check the status of sites. So, you need to write code for checking the status of a website.

You can choose to use either TCP or ICMP for your connections. The socket module is one to check out. You can also read Socket Programming in Python (Guide).

Through your chosen framework, be it the docopt, click, or argparse framework, you can add commands to allow users to add and remove sites from the list of sites to be checked.

The users should also be able to start the tool, stop it, and determine the intervals.

Since you’ll have to save the list of files to be checked, you can either save it in a file (just a list of sites) or use a SQLite database through the sqlite3 module.

Extra Challenge

The application can check for the connectivity status of sites and display the results to the command-line. But this will require the user to keep checking the command-line.

You can increase the challenge and implement a notification feature. The notification feature can be a sound played in the background to alert the user when a site’s status changes. You’ll need a database to store the previous status of a site. That’s the only way the tool can tell when the status changes.

Bulk File Rename Tool

Sometimes, you need to name all the files in a directory according to certain conventions. For example, you can name all the files in a directory with File0001.jpg, where the numbers increase based on the number of files in the directory. Doing this manually can be stressful and repetitive.

The Bulk File Rename Tool allows users to rename a large number of files, without having to manually rename files.

This saves users a lot of time. It spares them the trouble of having to do boring repetitive work and make mistakes. With the Bulk File Rename Tool, users can rename files in a couple of seconds without any mistakes.

Examples of Bulk File Rename Tools

Here are some implementations of the Bulk File Rename idea:

• Ren
• Rename

Technical Details

The main objective of this project idea is to rename files. So, the application needs to find a way to manipulate the target files. The os, sys, and shutil libraries will be useful for a large part of this project.

Your users will be able to rename all the files in the directory, using naming conventions. Therefore, they should be able to pass in the naming convention of choice. The regex module will help match the required naming patterns, if you understand how regex works.

A user may want to pass in a naming convention such as myfiles as part of the commands and expect that the tool renames all the files like myfilesXYZ, where XYZ is a number. They should also be able to choose the directory where the files to be renamed are.

Extra Challenge

The major challenge in this project is to rename all the files in a directory. But users may only need to name a certain number of files. To test your skills, you can implement a feature to allow users to choose the number of files to be renamed, instead of all the files.

Note that renaming only a certain number of files will require the tool to sort the files based on alphabetical order, time of file creation, or file size, depending on the user’s requirements.

Directory Tree Generator

Directories are like family trees: each directory has a particular relationship with other directories. No directories ever stays on its own, except an empty root directory.

When you’re working with files and directories, it is difficult to see the relationship between directories, as you can only see what exists in the current directory. You’re either using a file manager or working from the command-line.

With a Directory Tree Generator, you can see the relationship between files and directories like a tree or a map.

This makes it easier to understand the positioning of files and directories. A directory tree map is important when you’re explaining certain concepts, and a Directory Tree Generator makes it easier to get a visual representation of the file and directory relationships.

Examples of Directory Tree Generators

Here are some implementations of the Directory Tree Generator idea:

• Tree
• Dirtreex

Technical Details

The main objective of the Directory Tree Generator is to visualize the relationships between files and directories. The os library can be very useful in listing the files and directories in a chosen directory.

Using a framework such as docopt or argparse helps abstract a lot of stuff, allowing you to focus on writing code for the application’s logic.

In the application’s logic, you can decide how you want to represent files or directories. Using different colors is a brilliant way to go about it. You can use the colored library to print the files and directories in different colors.

You can also decide how deep you’d like the Directory Tree Generator to go. For example, if a directory has children directories twelve levels deep, you may decide to go only as deep as the fifth level.

If you wish, you can also let the user decide how deep they want the Directory Tree Generator to go.

Extra Challenge

Since the results of the generated directory tree will be on the command-line, you can go one step further. You can have the generator create images of the directory tree, so it’ll basically turn the text into an image.

You’ll find the pillow library useful for doing this.

Tips for Working on Projects

Working on projects can be difficult. That’s one reason why motivation and interest in a project will make it a less daunting task.

If you’re interested in a project, you’ll be able to put in the time to research as well as find libraries and tools that will help you with the project.

Here are some tips:

• Find a source of motivation
• Break the project into subtasks
• Do research on the subtasks
• Build each subtasks, one step at a time
• Reach out for help if you’re stuck
• Put the subtasks together

Conclusion

In this article, you’ve seen a couple of Python project ideas you may find interesting.

The project ideas cover a range of platforms. You saw project ideas for the Web, GUI, and Command-line platforms.

You can choose to build a project for different platforms. Using the URL Shortener as an example, you may choose to build one for the Web, GUI, or the Command-line.

Since you’re an intermediate Python developer, these projects can be quite challenging but interesting.

The best way to make a project happen is to just get started. In no time, you’ll be finished and discover how much you’ve benefited from working on a project!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

March 20, 2019 02:00 PM UTC

pythonwise

Speed: Default value vs checking for None

Python's dict has a get method. It'll either return an existing value for a given key or return a default value if the key is not in the dict. It's very tempting to write code like val = d.get(key, Object()), however you need to think about the performance implications. Since function arguments are evaluated before calling the function, this means the a new Object will be created regardless if it's in the dict or not. Let's see this affects performance.

get_default will create new Point every time and get_none will create only if there's no such object, it works since or evaluate it's arguments lazily and will stop once the first one is True.

First we'll try with a missing key:

In [1]: %run default_vs_none.py                                     
In [2]: locations = {}  # name -> Location 
In [3]: %timeit get_default(locations, 'carmen')
384 ns ± 2.56 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [4]: %timeit get_none(locations, 'carmen')
394 ns ± 1.61 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

March 20, 2019 11:50 AM UTC

leftmouseclickin

Plotting the average directional movement index values

Hello and welcome back, in this chapter we will continue developing our Forex and Stock application. We will create a method to plot the average directional movement index values. The average directional index (ADX) is a technical analysis indicator used by some traders to determine the strength of a trend. The trend can be either up or down, and this is shown by two accompanying indicators, the Negative Directional Indicator (-DI) and the Positive Directional Indicator (+DI).

Below is the modified version of the project with a plot adx button which will call the plot adx method after it has been clicked.

import json
from tkinter import *
import tkinter.ttk as tk
from alpha_vantage.foreignexchange import ForeignExchange
from alpha_vantage.techindicators import TechIndicators
from alpha_vantage.timeseries import TimeSeries
import matplotlib.pyplot as plt
from alpha_vantage.sectorperformance import SectorPerformances

win = Tk() # Create tk instance
win.title("Real Forex n Stock") # Add a title
win.resizable(0, 0) # Disable resizing the GUI
win.configure(background='white') # change window background color

selectorFrame = Frame(win, background="white") # create the top frame to hold base and quote currency combobox
selectorFrame.pack(anchor = "nw", pady = 2, padx=10)
currency_label = Label(selectorFrame, text = "Select base currency / quote currency :", background="white")
currency_label.pack(anchor="w") # the currency pair label

selector1Frame = Frame(win, background="white") # create the middle frame to hold base and quote currency combobox
selector1Frame.pack(anchor = "nw", pady = 2, padx=10)
stock_label = Label(selector1Frame, text = "Select Stock / Time Interval / Series type / Moving average type / Fast Period / Slow Period :", background="white")
stock_label.pack(anchor="w") # the stock label

curr1 = tuple() # the tuple which will be populated by base and quote currency
currency_list = ['AUD', 'BCH', 'BNB', 'BND', 'BTC', 'CAD', 'CHF', 'CNY', 'EOS', 'EUR', 'ETH', 'GBP', 'HKD', 'JPY', 'LTC', 'NZD', 'MYR', 'TRX', 'USD', 'USDT', 'XLM', 'XRP'] # major world currency pairs

# populate the combo box for both the base and quote currency
for key in currency_list:
    curr1 += (key, )

# populate the stock symbol tuple
f = open("stock.txt", "r")
curr2 = tuple()
for line in f.readlines():
    curr2 += (line.replace('\n', ''),)
f.close()

# Create a combo box for base currency
base_currency = StringVar() # create a string variable
based = tk.Combobox(selectorFrame, textvariable=base_currency)
based['values'] = curr1
based.pack(side = LEFT, padx=3)

# Create a combo box for quote currency
quote_currency = StringVar() # create a string variable
quote = tk.Combobox(selectorFrame, textvariable=quote_currency)
quote['values'] = curr1
quote.pack(side = LEFT, padx=3)

# Create a combo box for stock items
stock_symbol = StringVar() # create a string variable
stock = tk.Combobox(selector1Frame, textvariable=stock_symbol)
stock['values'] = curr2
stock.current(0)
stock.pack(side = LEFT, padx=3)

interval = tk.Combobox(selector1Frame)
interval['values'] = ('1min', '5min', '15min', '30min', '60min', 'daily', 'weekly', 'monthly')
interval.current(0)
interval.pack(side = LEFT, padx=3)

price_type = tk.Combobox(selector1Frame)
price_type['values'] = ('close', 'open', 'high', 'low')
price_type.current(0)
price_type.pack(side =LEFT, padx=3)

matype_type = tk.Combobox(selector1Frame, width=37)
matype_type['values'] = ('Simple Moving Average (SMA)', 'Exponential Moving Average (EMA)', 'Weighted Moving Average (WMA)', 'Double Exponential Moving Average (DEMA', 'Triple Exponential Moving Average (TEMA)', 'Triangular Moving Average (TRIMA', 'T3 Moving Average', 'Kaufman Adaptive Moving Average (KAMA)', ' MESA Adaptive Moving Average (MAMA)')
matype_type.current(0)
matype_type.pack(side =LEFT, padx=3)
mattype_list = ['Simple Moving Average (SMA)', 'Exponential Moving Average (EMA)', 'Weighted Moving Average (WMA)', 'Double Exponential Moving Average (DEMA', 'Triple Exponential Moving Average (TEMA)', 'Triangular Moving Average (TRIMA', 'T3 Moving Average', 'Kaufman Adaptive Moving Average (KAMA)', ' MESA Adaptive Moving Average (MAMA)']

# fill up the fast period and slow period combo boxes with integer ranging from 2 to 10,000
fa = tuple()
for i in range(2, 10001):
    fa += (i, )

fast_pe = tk.Combobox(selector1Frame)
fast_pe['values'] = fa
fast_pe.current(0)
fast_pe.pack(side=LEFT, padx=3)

slow_pe = tk.Combobox(selector1Frame)
slow_pe['values'] = fa
slow_pe.current(0)
slow_pe.pack(side=LEFT, padx=3)

# create text widget area
s = StringVar() # create string variable which will be used to fill up the Forex data
# create currency frame and text widget to display the incoming forex data
currencyFrame = Frame(win)
currencyFrame.pack(side=TOP, fill=X)
currency = Label(currencyFrame)
currency.pack(fill=X)
text_widget = Text(currency, fg='white', background='black')
text_widget.pack(fill=X)
s.set("Click the find button to find out the currency exchange rate")
text_widget.insert(END, s.get())

buttonFrame = Frame(win) # create a bottom frame to hold the find button
buttonFrame.pack(side = BOTTOM, fill=X, pady = 6, padx=10)

# first get the api key and secret from the file
f = open("alpha.txt", "r")
api_key = f.readline()
f.close()
api_key = api_key.replace('\n', '')

def get_exchange_rate(): # this method will display the incoming forex data after the api called

    try:
        cc = ForeignExchange(key= api_key)
        from_ = based.get()
        to_ = quote.get()

        countVar = StringVar()  # use to hold the character count
        text_widget.tag_remove("search", "1.0", "end")  # cleared the hightlighted currency pair

        if(from_ != '' and to_ != '' and from_ != to_):
            data, _ = cc.get_currency_exchange_rate(from_currency=from_, to_currency=to_)
            exchange_rate = dict(json.loads(json.dumps(data)))
            count = 1
            sell_buy = str(count) + ".) Pair : " + exchange_rate['1. From_Currency Code'] + "(" + exchange_rate['2. From_Currency Name'] + ")" + " / " + exchange_rate['3. To_Currency Code']+"(" + exchange_rate['4. To_Currency Name'] + ") : "  + str(exchange_rate['5. Exchange Rate']) + '\n'
            text_widget.delete('1.0', END)  # clear all those previous text first
            s.set(sell_buy)
            text_widget.insert(INSERT, s.get())  # display forex rate in text widget
            pos = text_widget.search(from_, "1.0", stopindex="end", count=countVar)
            text_widget.tag_configure("search", background="green")
            end_pos = float(pos) + float(0.7)
            text_widget.tag_add("search", pos, str(end_pos))  # highlight the background of the searched currency pair
            pos = float(pos) + 2.0
            text_widget.see(str(pos))

    except:
        print("An exception occurred")

def plot_stock_echange():

    try:
        stock_symbol_text = stock.get() # get the selected symbol
        if(stock_symbol_text!= ''):
            ts = TimeSeries(key=api_key, output_format='pandas')
            data, meta_data = ts.get_intraday(symbol=stock_symbol_text, interval='1min', outputsize='full')
            data['4. close'].plot()
            stock_title = 'Intraday Times Series for the ' + stock_symbol_text + ' stock (1 min)'
            plt.title(stock_title)
            plt.show()
    except:
        print("An exception occurred")

def plot_stock_technical():

    try:
        stock_symbol_text = stock.get() # get the selected stock symbol
        if(stock_symbol_text!= ''):

            ti = TechIndicators(key=api_key, output_format='pandas')
            data, meta_data = ti.get_bbands(symbol=stock_symbol_text, interval=interval.get(), series_type=price_type.get(), matype=mattype_list.index(matype_type.get()), time_period=int(interval.get().replace('min', '')))
            data.plot()
            stock_title = 'BBbands indicator for ' + stock_symbol_text + ' ' + interval.get()
            plt.title(stock_title)
            plt.show()
    except:
        print("An exception occurred")

def plot_op(): # plot the Absolute price oscillator (APO)

    try:
        stock_symbol_text = stock.get() # get the selected stock symbol
        if(stock_symbol_text!= ''):

            ti = TechIndicators(key=api_key, output_format='pandas')
            data, meta_data = ti.get_apo(symbol=stock_symbol_text, interval=interval.get(), series_type=price_type.get(), matype=mattype_list.index(matype_type.get()), fastperiod = fast_pe.get(), slowperiod= slow_pe.get())
            data.plot()
            stock_title = 'Absolute Price Oscillator (APO) for ' + stock_symbol_text + ' ' + interval.get()
            plt.title(stock_title)
            plt.show()
    except ValueError:
        print("An exception occurred")

def plot_adxr(): # plot the average directional movement index rating

    try:
        stock_symbol_text = stock.get() # get the selected stock symbol
        if(stock_symbol_text!= ''):

            ti = TechIndicators(key=api_key, output_format='pandas')
            data, meta_data = ti.get_adxr(symbol=stock_symbol_text, interval=interval.get(), time_period=int(interval.get().replace('min', '')))
            data.plot()
            stock_title = 'Average directional movement index rating for ' + stock_symbol_text + ' at ' + interval.get()
            plt.title(stock_title)
            plt.show()
    except ValueError:
        print("An exception occurred")

def plot_adx(): # plot the average directional movement index

    try:
        stock_symbol_text = stock.get() # get the selected stock symbol
        if(stock_symbol_text!= ''):

            ti = TechIndicators(key=api_key, output_format='pandas')
            data, meta_data = ti.get_adx(symbol=stock_symbol_text, interval=interval.get(), time_period=int(interval.get().replace('min', '')))
            data.plot()
            stock_title = 'Average directional movement index for ' + stock_symbol_text + ' at ' + interval.get()
            plt.title(stock_title)
            plt.show()
    except ValueError:
        print("An error exception occurred")

def plot_sector_performance():

    sp = SectorPerformances(key=api_key, output_format='pandas')
    data, meta_data = sp.get_sector()
    data['Rank A: Real-Time Performance'].plot(kind='bar')
    plt.title('Real Time Performance (%) per Sector')
    plt.tight_layout()
    plt.grid()
    plt.show()

def plot_ad():

    try:
        stock_symbol_text = stock.get()  # get the selected stock symbol
        if (stock_symbol_text != ''):
            ti = TechIndicators(key=api_key, output_format='pandas')
            data, meta_data = ti.get_ad(symbol=stock_symbol_text, interval=interval.get())
            data.plot()
            stock_title = 'Chaikin A/D line values for ' + stock_symbol_text + ' ' + interval.get()
            plt.title(stock_title)
            plt.show()
    except:
        print("An exception occurred")

action_vid = tk.Button(buttonFrame, text="Calculate Exchange Rate", command=get_exchange_rate) # button used to find out the exchange rate of currency pair
action_vid.pack(side=LEFT, padx=2)

action_stock_plot = tk.Button(buttonFrame, text="Plot Stock", command=plot_stock_echange) # button used to plot the intra-minute stock value
action_stock_plot.pack(side=LEFT, padx=2)
action_technical_plot = tk.Button(buttonFrame, text="Plot Technical", command=plot_stock_technical) # button used to plot the 60 minutes stock technical value
action_technical_plot.pack(side=LEFT, padx=2)
action_sector_plot = tk.Button(buttonFrame, text="Plot Sector Performance", command=plot_sector_performance) # button used to plot the sector performance graph
action_sector_plot.pack(side=LEFT, padx=2)
action_ad_plot = tk.Button(buttonFrame, text="Plot AD Line", command=plot_ad) # button used to plot the A/D line graph
action_ad_plot.pack(side=LEFT, padx=2)
action_ad_op = tk.Button(buttonFrame, text="Plot APO Line", command=plot_op) # button used to plot the APO line graph
action_ad_op.pack(side=LEFT, padx=3)
action_adxr = tk.Button(buttonFrame, text="Plot ADXR Line", command=plot_adxr) # button used to plot the average directional movement index rating
action_adxr.pack(side=LEFT, padx=3)
action_adx = tk.Button(buttonFrame, text="Plot ADX Line", command=plot_adx) # button used to plot the average directional movement index
action_adx.pack(side=LEFT, padx=3)
win.iconbitmap(r'ico.ico')
win.mainloop()

After running the above program below is what we will get.

Plotting the #stock average directional movement index values with #python program pic.twitter.com/iuyyGvBxjW

— TechLikin (@ChooWhei) March 20, 2019

Like, share or follow me on Twitter.

March 20, 2019 11:17 AM UTC

Shyama Sankar Vellore

Python Sets: Cheat Sheet

A cheat sheet for sets in Python. What is the 'set' datatype in Python? How is it used?

Jump to the cheat sheet

Key points

Cheat sheet

Useful resources and references

March 20, 2019 07:19 AM UTC

March 19, 2019

Codementor

Clean up and remove a Python3 homebrew install

tl;dr how to clean up your Python3 homebrew install

March 19, 2019 08:47 PM UTC

PyCoder’s Weekly

Issue #360 (March 19, 2019)

#360 – MARCH 19, 2019
View in Browser »

Automated Code Reviews for Python

Take the hassle out of code reviews—Codacy flags errors so you can fix them quickly. Address security concerns, code duplication, code complexity and drops in coverage, directly from your workflow. Click here to get started →
CODACY sponsor

Discussions

Python Jobs

Brought to you by Indeed Prime

sponsor

Apply today to see what job opportunities are waiting for you!

Machine Learning and Data Science Developer (Austin, TX)

Protection Engineering Consultants LLC

Lead Python Developer (Toronto, Canada)

Kognitiv

Senior Systems Engineer (Hamilton, Canada)

Preteckt

Python Software Engineer (Berlin, Germany)

Wooga

Computer Science Teacher (Pasadena, CA)

ArtCenter College of Design

Senior Python Engineer (New York, NY)

15Five

More Python Jobs >>>

Articles & Tutorials

Take ActiveState’s ~7 Min Survey for a Chance to Win a Lego Star Wars TIE Fighter

We want to know your pain points building & programming open source languages. We’ve got 3 cool prizes too. Click to speed through this ~7min survey →
ACTIVESTATE sponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #360.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

March 19, 2019 07:30 PM UTC

Mike Driscoll

How to Distribute a wxPython Application

Let’s say you finished up a wonderful GUI application using wxPython. How do you share it with the world? This is always the dilemma when you finish an amazing program. Fortunately, there are several ways you can share your code. If you want to share your code with other developers, than Github or a similar website is definitely a good way to do. I won’t be covering using Git or Mercurial here. Instead what you will learn here is how to turn your application into an executable.

By turning your code into an executable, you can allow a user to just download the binary and run it without requiring them to download Python, your source code and your dependencies. All of those things will be bundled up into the executable instead.

There are many tools you can use to generate an executable:

• py2exe
• py2app
• PyInstaller
• cx_Freeze
• bbfreeze
• Nuitka
• Briefcase

You will be using PyInstaller in this tutorial. The main benefit to using PyInstaller is that it can generate executables for Windows, Mac and Linux. Note that it does not support cross-compiling. What that means is that you cannot run PyInstaller on Linux to create a Windows executable. Instead, PyInstaller will only create an executable for the OS that it is ran on. In other words, if you run PyInstaller on Windows, it will create a Windows executable only.

---

Installing PyInstaller

Installing the PyInstaller package is nice and straightforward. All you need is pip.

Here is how you would install PyInstaller to your system Python:

pip install pyinstaller

You could also install PyInstaller to a virtual Python environment using Python’s venv module or the virtualenv package.

---

Generating an Executable

The nice thing about PyInstaller is that it is very easy to use out of the box. All you need to do is run the `pyinstaller` command followed by the path to the main file of the application that you want to convert to an executable.

Here is a non-working example:

pyinstaller path/to/main/script.py

If the PyInstaller application is not found, you may have to specify a full path to it. By default, PyInstaller installs to Python’s **Scripts** sub-folder, which is going to be in your system Python folder or in your virtual environment.

Let’s take one of the simple applications from my upcoming book and turn it into an executable. For example, you could use image_viewer_slideshow.py from chapter 3:

# image_viewer_slideshow.py
 
import glob
import os
import wx
 
class ImagePanel(wx.Panel):
 
    def __init__(self, parent):
        super().__init__(parent)
        self.max_size = 240
        self.photos = []
        self.current_photo = 0
        self.total_photos = 0
        self.layout()
 
        self.slideshow_timer = wx.Timer(self)
        self.Bind(wx.EVT_TIMER, self.on_next, self.slideshow_timer)
 
    def layout(self):
        """
        Layout the widgets on the panel
        """
 
        self.main_sizer = wx.BoxSizer(wx.VERTICAL)
        btn_sizer = wx.BoxSizer(wx.HORIZONTAL)
 
        img = wx.Image(self.max_size, self.max_size)
        self.image_ctrl = wx.StaticBitmap(self, wx.ID_ANY, 
                                             wx.Bitmap(img))
        self.main_sizer.Add(self.image_ctrl, 0, wx.ALL|wx.CENTER, 5)
        self.image_label = wx.StaticText(self, label="")
        self.main_sizer.Add(self.image_label, 0, wx.ALL|wx.CENTER, 5)
 
        btn_data = [("Previous", btn_sizer, self.on_previous),
                    ("Slide Show", btn_sizer, self.on_slideshow),
                    ("Next", btn_sizer, self.on_next)]
        for data in btn_data:
            label, sizer, handler = data
            self.btn_builder(label, sizer, handler)
 
        self.main_sizer.Add(btn_sizer, 0, wx.CENTER)
        self.SetSizer(self.main_sizer)
 
    def btn_builder(self, label, sizer, handler):
        """
        Builds a button, binds it to an event handler and adds it to a sizer
        """
        btn = wx.Button(self, label=label)
        btn.Bind(wx.EVT_BUTTON, handler)
        sizer.Add(btn, 0, wx.ALL|wx.CENTER, 5)
 
    def on_next(self, event):
        """
        Loads the next picture in the directory
        """
        if not self.photos:
            return
 
        if self.current_photo == self.total_photos - 1:
            self.current_photo = 0
        else:
            self.current_photo += 1
        self.update_photo(self.photos[self.current_photo])
 
    def on_previous(self, event):
        """
        Displays the previous picture in the directory
        """
        if not self.photos:
            return
 
        if self.current_photo == 0:
            self.current_photo = self.total_photos - 1
        else:
            self.current_photo -= 1
        self.update_photo(self.photos[self.current_photo])
 
    def on_slideshow(self, event):
        """
        Starts and stops the slideshow
        """
        btn = event.GetEventObject()
        label = btn.GetLabel()
        if label == "Slide Show":
            self.slideshow_timer.Start(3000)
            btn.SetLabel("Stop")
        else:
            self.slideshow_timer.Stop()
            btn.SetLabel("Slide Show")
 
    def update_photo(self, image):
        """
        Update the currently shown photo
        """
        img = wx.Image(image, wx.BITMAP_TYPE_ANY)
        # scale the image, preserving the aspect ratio
        W = img.GetWidth()
        H = img.GetHeight()
        if W > H:
            NewW = self.max_size
            NewH = self.max_size * H / W
        else:
            NewH = self.max_size
            NewW = self.max_size * W / H
        img = img.Scale(NewW, NewH)
 
        self.image_ctrl.SetBitmap(wx.Bitmap(img))
        self.Refresh()
 
    def reset(self):
        img = wx.Image(self.max_size,
                       self.max_size)
        bmp = wx.Bitmap(img)
        self.image_ctrl.SetBitmap(bmp)
        self.current_photo = 0
        self.photos = []
 
 
class MainFrame(wx.Frame):
 
    def __init__(self):
        super().__init__(None, title='Image Viewer',
                                        size=(400, 400))
        self.panel = ImagePanel(self)
        self.create_toolbar()
        self.Show()
 
    def create_toolbar(self):
        """
        Create a toolbar
        """
        self.toolbar = self.CreateToolBar()
        self.toolbar.SetToolBitmapSize((16,16))
 
        open_ico = wx.ArtProvider.GetBitmap(
            wx.ART_FILE_OPEN, wx.ART_TOOLBAR, (16,16))
        openTool = self.toolbar.AddTool(
            wx.ID_ANY, "Open", open_ico, "Open an Image Directory")
        self.Bind(wx.EVT_MENU, self.on_open_directory, openTool)
 
        self.toolbar.Realize()
 
    def on_open_directory(self, event):
        """
        Open a directory dialog
        """
        with wx.DirDialog(self, "Choose a directory",
                          style=wx.DD_DEFAULT_STYLE) as dlg:
 
            if dlg.ShowModal() == wx.ID_OK:
                self.folderPath = dlg.GetPath()
 
                photos = glob.glob(os.path.join(self.folderPath, '*.jpg'))
                self.panel.photos = photos
                if photos:
                    self.panel.update_photo(photos[0])
                    self.panel.total_photos = len(photos)
                else:
                    self.panel.reset()
 
 
if __name__ == '__main__':
    app = wx.App(redirect=False)
    frame = MainFrame()
    app.MainLoop()

If you wanted to turn it into an executable, you would run the following:

pyinstaller image_viewer_slideshow.py

Make sure that when you run this command, your current working directory is the one that contains the script you are converting to an executable. PyInstaller will be creating its output in whatever the current working directory is.

When you run this command, you should see something like this in your terminal:

PyInstaller will create two folders in the same folder as the script that you are converting called **dist** and **build**. The **dist** folder is where you will find your executable if PyInstaller completes successfully. There will be many other files in the **dist** folder besides your executable. These are files that are required for your executable to run.

Now let’s try running your newly created executable. When I ran my copy, I noticed that a terminal / console was appearing behind my application.

Image Viewer with Console in Background

This is normal as the default behavior of PyInstaller is to build your application as if it were a command-line application, not a GUI.

You will need to add the –noconsole flag to remove the console:

pyinstaller image_viewer_slideshow.py --noconsole

Now when you run the result, you should no longer see a console window appearing behind your application.

It can be complicated to distribute lots of files, so PyInstaller has another command that you can use to bundle everything up into a single executable. That command is `–onefile`. As an aside, a lot of the commands that you use with PyInstaller have shorter aliases. For example, there is a shorter alias for `–noconsole` that you can also use called: -w. Note the single dash in `-w`.

So let’s take that information and have PyInstaller create a single file executable with no console:

[python] pyinstaller image_viewer_slideshow.py --onefile -w

You should now have just one file in the dist folder.

---

The spec file

PyInstaller has the concept of specification files. They are kind of like a setup.py script, which is something that you use with Python’s distutils. These spec files tell PyInstaller how to build your executable. PyInstaller will generate one for you automatically with the same name as the passed in script, but with a .spec extension. So if you passed in image_viewer_slideshow.py, then you should see a image_viewer_slideshow.spec file after running PyInstaller. This spec file will be created in the same location as your application file.

Here is the contents of the spec file that was created from the last run of PyInstaller above:

# -*- mode: python -*-
 
block_cipher = None
 
 
a = Analysis(['image_viewer.py'],
             pathex=['C:\\Users\\mdriscoll\\Documents\\test'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          [],
          name='image_viewer',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          runtime_tmpdir=None,
          console=False )

While PyInstaller worked fine with the image viewer example, you may find that it won’t work out of the box if you had other dependencies, such as NumPy or Pandas. If you run into issues with PyInstaller, it has very verbose logs that you can use to help you figure out the issue. One good location is the `build/cli/warn-cli.txt` file. You may also want to rebuild without the `-w` command so that you can see what is being printed to stdout in the console window.

There are also options for changing the log level during building that may help you uncover issues.

If none of those work, try Google or go to PyInstaller’s support page and get help there.

---

Creating Executables for Mac

While the same commands should work on Mac OSX as it does on Windows, I found that I needed to run the following command to generate a working executable:

pyinstaller image_viewer_slideshow.py --windowed

The output that PyInstaller generates will be slightly different and the result is an application file.

Another popular option for generating applications on Mac is a Python package called py2app.

---

Creating Executables for Linux

For Linux, it is usually recommended that you build the executable with an old version of glibc because the newer glibc versions are backwards compatible. By building with an old version of Linux, you can usually target a wider variety of Linux versions. But your mileage may vary.

After the files are generated, you can just tar them up into a gzipped tarball (.tax.gz). You could even using the archiving application you created in this book to do that for you, if you wanted.

An alternative would be to learn how to create a .deb or related file that most Linux versions can install.

---

Learning More About PyInstaller

This article is not meant to be an in-depth guide to PyInstaller. It will likely change much faster than wxPython, so it is recommended that you read the documentation for PyInstaller instead. It will always be the most up-to-date location to get the information you need on the project.

---

What About Installers?

Windows users know that most of the time you have an installer application that you can run to install your application on your computer and put some shortcuts here and there. There are several useful free programs that you can use to create a Windows Installer as well as some paid ones

Here are the two freeware applications I see mentioned the most:

• NSIS
• Inno Setup

I have used Inno Setup to create a Windows installer on several occasions. It is easy to use and requires only a little reading of its documentation to get it working. I haven’t used NSIS before, but I suspect it is quite easy to use as well.

Let’s use Inno Setup as an example and see how to generate an installer with it.

---

Creating an Installer with Inno Setup

Inno Setup is a nice freeware application that you can use to create professional looking installer programs. It works on most versions of Windows. I personally have used it for quite a few years. While Inno Setup is not open source, it is still a really nice program. You will need to download and install it from there website.

Once installed, you can use this tool to create an installer for the executable you created earlier in this chapter.

To get started, just run Inno Setup and you should see the following:

Inno Setup’s Startup Page

While Inno Setup defaults to opening an existing file, what you want to do is choose the second option from the top: “Create a new script file using the Script Wizard”. Then press **OK**.

You should now see the first page of the Inno Setup Script Wizard. Just hit **Next** here since there’s nothing else you can really do.

Now you should see something like this:

Inno Setup Script Wizard Application Information Page

This is where you enter your applications name, its version information, the publisher’s name and the application’s website. I pre-filled it with some examples, but you can enter whatever you want to here.

Go ahead and press Next and you should see page 3:

Inno Setup Script Wizard Application Folder Page

This page of the wizard is where you can set the application’s install directory. On Windows, most applications install to **Program Files**, which is also the default here. This is also where you set the folder name for your application. This is the name of the folder that will appear in Program Files. Alternatively, you can check the box at the bottom that indicates that your application doesn’t need a folder at all.

Let’s go to the next page:

Inno Setup Script Wizard Application Files Page

Here is where you will choose the main executable file. In this case, you want to choose the executable you created with PyInstaller. If you didn’t create the executable using the –onefile flag, then you can add the other files using the Add file(s)… button. If your application requires any other special files, like a SQLite database file or images, this is also where you would want to add them.

By default, this page will allow the user to run your application when the installer finishes. A lot of installers do this, so it’s actually expected by most users.

Let’s continue:

Inno Setup Script Wizard Application Shortcuts Page

This is the Application Shortcuts page and it allows you to manage what shortcuts are created for your application and where they should go. The options are pretty self-explanatory. I usually just use the defaults, but you are welcome to change them however you see fit.

Let’s find out what’s on the documentation page:

Inno Setup Script Wizard Application Documentation Page

The Documentation Page of the wizard is where you can add your application’s license file. For example, if you were putting out an open source application, you can add the GPL or MIT or whatever license file you need there. If this were a commercial application, this is where you would add your End-Users License Agreement (EULA) file.

Let’s see what’s next:

Inno Setup Script Wizard Setup Languages Page

Here you can set up which setup languages should be included. Inno Setup supports quite a few languages, with English as the default choice.

Now let’s find out what compiler settings are:

Inno Setup Script Wizard Compiler Settings Page

The Compiler Settings page let’s you name the output setup file, which defaults to simply **setup**. You can set the output folder here, add a custom setup file icon and even add password protection to the setup file. I usually just leave the defaults alone, but this is an opportunity to add some branding to the setup if you have a nice icon file handy.

The next page is for the preprocessor:

Inno Setup Script Wizard Preprocessor Page

The preprocessor is primarily for catching typos in the Inno Setup script file. It basically adds some helpful options at compile time to your Inno Setup script.

Check out the documentation for full details.

Click Next and you should see the last page of the wizard:

Inno Setup Script Wizard End Page

Click Finish and Inno Setup will generate an Inno Setup Script (.iss) file. When it is finished, it will ask you if you would like to compile the file.

Go ahead and accept that dialog and you should see the following:

Inno Setup Script

This is the Inno Setup Script editor with your newly generated script pre-loaded into it. The top half is the script that was generated and the bottom half shows the compiler’s output. In this screenshot, it shows that the setup file was generated successfully but it also displays a warning that you might want to rename the setup file.

At this point, you should have a working installer executable that will install your program and any files it depends on to the right locations. It will also create shortcuts in the Windows Start menu and whichever other locations you specified in the wizard.

The script file itself can be edited. It is just a text file and the syntax is well documented on Inno Setup’s website.

---

Code Signing

Windows and Mac OSX prefer that applications are signed by a corporation or the developer. Otherwise you will see a warning that you are using an unsigned piece of code or software. The reason this matters is that it protects your application from being modified by someone else. You can think of code signing as a kind of embedded MD5 hash in your application. A signed application can be traced back to whomever signed it, which makes it more trust-worthy.

If you want to sign code on Mac OSX, you can use XCode

Windows has several options for signing their code. Here is a URL for getting your application certified for Windows

You can also purchase a certificate from various companies that specialize in code signing, such as digicert.

There is also the concept of self-signed certificates, but that is not for production or for end users. You would only self-sign for internal testing, proof-of-concept, etc. You can look up how to do that on your own.

---

Wrapping Up

You have now learned how to generate executables using PyInstaller on Windows, Mac and Linux. The command to generate the executable is the same across all platforms. While you cannot create a Windows executable by running PyInstaller on Linux, it is still quite useful for creating executable for the target operating system.

You also learned how to use Inno Setup to create an installer for Windows. You can now use these skills to create executables for your own applications or for some of the other applications that you created in this book!

---

Further Reading

• A bbfreeze Tutorial – Build a Binary Series!
• Packaging wxPyMail for Distribution

March 19, 2019 05:15 PM UTC

Django Weblog

2018 Malcolm Tredinnick Memorial Prize awarded to Kojo Idrissa

The Board of the Django Software Foundation is pleased to announce that the 2018 Malcolm Tredinnick Memorial Prize has been awarded to Kojo Idrissa.

Kojo has been active in the Django community since at least 2015, if not earlier. He's been a DjangoCon US organizer since 2016, former DEFNA board member, and current DEFNA North American Ambassador.

Kojo has hosted an orientation for first-time DjangoCon US attendees for the last several years, which could not be a better example of Malcolm's friendly spirit to new users.

Ken Whitesell, who nominated Kojo, also noted many Kojo's other contributions:

Kojo is a very active member of the weekly CodeNewbie chats. Hosts the DjangoCon new-user orientation session. Very visible presence at DjangoCon, always seems to be focused on ensuring first time attendees have the best possible experience.

The other nominees this year were:

• Anna Makarudze
• Daniel Joey Darko
• Humphrey Butau
• Jani Tiainen
• Jeff Triplett
• Kenneth Love
• Simon Charette

Every year we receive many nominations and it's always hard to pick the winner. In fact, some have been nominated in multiple years. Malcolm would be very proud of the legacy he has fostered in our community!

Congratulations Kojo on the well deserved honor!

March 19, 2019 04:04 PM UTC

Real Python

Python Decorators 101

In this course on Python decorators, you’ll learn what they are and how to create and use them. Decorators provide a simple syntax for calling higher-order functions in Python. By definition, a decorator is a function that takes another function and extends the behavior of the latter function without explicitly modifying it.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

March 19, 2019 02:00 PM UTC

Abhijeet Pal

Starting A Django Project

Django is one of the most popular open source full-stack web development Framework written in Python. Django is a framework for perfectionists with deadlines, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel.

To know more about Django read: Django – Web Framework For Perfectionists

In this article, we will go over the steps for creating and running a Django Project.

Creating A Django Project

A project is a collection of settings for an instance of Django including database configuration, Django-specific options, and application-specific settings.

Before creating a Django project make sure you have Python and Django installed in your machine if not go through How To Install Django

It is recommended to create projects in virtual environments though it’s optional using them will save you from a lot of headaches in the future. Read the following article to know why virtual environments are so important: How To A Create Virtual Environment for Python

Note that, I am assuming that you have activated your virtual environment where your Django installation exists. Next, run the below command.

django-admin startproject mysite

This command will invoke the django-admin.py script, which will set up a new Django project called mysite. A Django project name can be composed of numbers, letters, or underscores. A project name cannot start with a number, it can only start with a letter or underscore in addition special characters and spaces aren’t allowed anywhere in a project name.

You’ll now notice within your workspace there is a directory set to the name of your new project, mysite. Inside the directory, there is another directory with the same name as the project and a python script named manage.py

Inside the second level project directory, there are another 4 Python scripts. So the default Django project can be illustrated as follows.

mysite/
   manage.py
   mysite/
      __init__.py
      settings.py
      urls.py
      wsgi.py

__init__.py –  a blank Python script whose presence indicates to the Python interpreter that the directory is a Python package. This file allows Python packages to be imported from the directories they are present.

settings.py – Contains the configuration settings for the Django project.

urls.py – Contains URL patterns for the Django project.

wsgi.py – Contains WSGI configuration properties for the Django project. Basically, Python script used to help run your development server and deploy your project to a production environment.

manage.py –  a command-line utility that lets you interact with your Django project in many ways. This script is responsible for all project specific tasks.

Note that, there are two directories with the project name, this might seem confusing at first however you can always change the name of the outer directory which occasionally referred to as the Base Directory. The second level project directory is hardcoded in some of the Django specific files so better not mess with it.

Running A Django Project

Now that we have created a Django project we can see it in action in the browser. Django comes with a built-in web server for development. Navigate to the Base directory where manage.py is, and run the below command.

python manage.py runserver

Ignore all the migration errors for the moment, Now open your preferred browser and go to http://127.0.0.1:8000/ If everything went right you should see the default the page of Django.

To stop the server press ctrl + c in the terminal window. Running the application on a different port is also possible you just need to pass the port number along with the command.

# Run the development server on the local address and port 4345 (http://127.0.0.1:4345/)
python manage.py runserver 4345

# Run the dev server on the 96.126.104.88 address and port 80 (http://96.126.104.88/)
python manage.py runserver 96.126.104.88:80

The post Starting A Django Project appeared first on Django Central.

March 19, 2019 12:11 PM UTC

Shyama Sankar Vellore

Python Tuples: Cheat Sheet

A cheat sheet for tuples in Python. What are tuples? How are they used in Python?

Useful Resources and References

March 19, 2019 06:56 AM UTC

• • RSS feed
  • Titles Only
  • Powered by Planet!
• Other Python Planets
  • Python Summer of Code
  • Planet Python Francophone
  • Planet Python Argentina
  • Planet Python Japan
  • Planet Python Brasil
  • Planet Python Indonesia
  • Planet Python Poland
• Python Libraries
  • PySoy
  • SciPy
  • SymPy
  • Twisted
• Python/Web Planets
  • CherryPy
  • Django Community
  • Plone
  • Turbogears
• Other Languages
  • Haskell
  • Lisp
  • Parrot
  • Perl
  • Ruby
• Databases
  • MySQL
  • PostgreSQL
• Subscriptions
  • [OPML feed]
  • "Control F'd"
  • "Fredrik Håård's Blaag"
  • "Menno's Musings"
  • "Michael J.T. O'Kelly"
  • "Morphex's Blogologue"
  • "Speno's Pythonic Avocado"
  • "William's Journal"
  • 2degrees
  • 4teamwork
  • A. Jesse Jiryu Davis
  • ABlog for Sphinx
  • AWeber Engineering
  • Aahz
  • Abhijeet Pal
  • Abu Ashraf Masnun
  • Adam Pletcher
  • Agendaless Consulting
  • Al-Ahmadgaid Asaad
  • Albert Hopkins
  • Alec Munro
  • Alex Clark
  • Alex Grönholm
  • Alex Morozov
  • Alexander Limi
  • Alexandre Conrad
  • Alexandre Vassalotti
  • Alexey Evseev
  • Allison Kaptur
  • Amit Saha
  • Amjith Ramanujam
  • AmvTek
  • Anarcat
  • Anastasios Hatzis
  • Anatoly Techtonik
  • Andre Roberge
  • Andrea Grandi
  • Andrew Dalke
  • Andriy Kornatskyy
  • Andy Dustman
  • Andy R. Terrel
  • Ankit Chandawala
  • Anna Martelli Ravenscroft
  • Anthony Baxter
  • Anton Belyaev
  • Anton Bobrov
  • Anwesha Das
  • AppNeta Blog
  • Armin Ronacher
  • Arnav Khare
  • Artem Golubin
  • Artem Rys
  • Ashish Vidyarthi
  • Astro Code School
  • Atul Varma -- Toolness
  • Audrey Roy Greenfeld
  • Automating OSINT
  • Baiju Muthukadan
  • Bajusz Tamás
  • Balthazar Rouberol
  • BangPypers
  • BeDjango
  • Ben Bass
  • Ben Rousch
  • Ben Tappin
  • Benjamin Peterson
  • Benji York
  • Bertrand Mathieu
  • Bhishan Bhandari
  • Bill Ward / AdminTome
  • BioPython News
  • Bit of Cheese
  • Blue Yonder Tech
  • BlueBream
  • Bojan Mihelac
  • Brad Lucas
  • Brandon Rhodes
  • BreadcrumbsCollector
  • Brendan Scott
  • Brett Cannon
  • Brian Harring
  • Brian Okken
  • Bruno Oliveira
  • Bruno Rocha
  • Caktus Consulting Group
  • Calvin Cheng
  • Calvin Spealman
  • Carl Chenet
  • Carl Trachte
  • Carlos Eduardo de Paula
  • Carlos de la Guardia
  • Casey Duncan
  • Catherine Devlin
  • Checking and Sharing
  • Chris Hager
  • Chris Leary
  • Chris McAvoy
  • Chris McDonough
  • Chris Miles
  • Chris Miller
  • Chris Mitchell
  • Chris Moffitt
  • Chris Rose
  • Chris Warrick
  • Christian Barra
  • Christian Heimes
  • Christian Scholz
  • Christoph Zwerschke
  • Christopher Lenz
  • Chuck Thier
  • ClusterHQ
  • Cobe.io
  • CodeSnipers
  • Codementor
  • Coding Diet
  • Collin Winter
  • Continuum Analytics Blog
  • Continuum Analytics News
  • Continuum Blog
  • Core Software
  • Corey Goldberg
  • Corey Oordt
  • Cormoran Project
  • Craig Kerstiens
  • Cross-Platform Command Line Tools
  • CubicWeb
  • Curtis Miller
  • DSPIllustrations.com
  • DaPythonista
  • Daily Tech Video (Python)
  • Dalius Dobravolskas
  • Dallas Fort Worth Pythoneers
  • Damián Avila
  • Dan Crosta
  • Dan Stromberg
  • Dan Yeaw
  • Daniel Bader
  • Daniel Nouri
  • Daniel Roy Greenfeld
  • Dariusz Suchojad
  • Data Community DC
  • Data School
  • DataCamp
  • Dataquest
  • Dave Beazley
  • David Goodger
  • David Grant
  • David J C Beach
  • David MacIver
  • David Malcolm
  • David Marte
  • David Stanek
  • David Szotten
  • Davide Moro
  • Davy Mitchell
  • Davy Wybiral
  • DeGizmo
  • Denis Kurov
  • Derrick Petzold
  • Dethe Elza
  • Diego Garcia
  • Django Weblog
  • Django Weekly
  • Djangostars
  • Doing Math with Python
  • Doug Hellmann
  • Doug Latornell
  • Dougal Matthews
  • Douglas Napoleone
  • DreamHost
  • Duncan McGreggor
  • Dusty Phillips
  • EasyGUI
  • Ed Crewe
  • Ed Taekema
  • Edward K. Ream
  • Eko S. Wibowo
  • Eli Bendersky
  • Eniram Ltd.
  • Enthought
  • Eray Özkural (examachine)
  • Eric Holscher
  • Erik Marsja
  • Etienne Desautels
  • EuroPython
  • EuroPython Society
  • Even Wiik Thomassen
  • Evennia
  • Experienced Django
  • Fabio Zadrozny
  • Fazal Majid
  • Filip Wasilewski
  • Filipe Saraiva
  • Flavio Coelho
  • Flavio Percoco
  • Floris Bruynooghe
  • Francisco Souza
  • Frank Wierzbicki
  • François Dion
  • François Marier
  • Frederik Rietdijk
  • Fredrik Lundh
  • From Python Import Podcast
  • Full Stack Python
  • Gaël Varoquaux
  • Gabriel Pettier
  • Galvanize
  • Gary Bernhardt
  • Gary Wilson
  • Geek Scrap
  • Georg Brandl
  • Georges Dubus
  • Ghaandee on IT
  • Giampaolo Rodola
  • Giulio Fidente
  • Glenn Franxman
  • Glyph Lefkowitz
  • Go Deh
  • GoDjango
  • Gocept Weblog
  • Godson Gera
  • Gonçalo Margalho
  • Graeme Cross
  • Graham Dumpleton
  • Graham Wheeler
  • Grant Baillie
  • Grant Rettke
  • Greg Taylor
  • Gridarrow
  • Grig Gheorghiu
  • Grzegorz Śliwiński
  • Guido van Rossum
  • Guilherme Toti
  • Gustavo Narea
  • Gustavo Niemeyer
  • Guyon Moree
  • Gökhan Sever
  • Hany Fahim
  • Hector Garcia
  • Heikki Toivonen
  • Hernan Grecco
  • Hilary Mason
  • Holger Krekel
  • Holger Peters
  • Hynek Schlawack
  • IO Digital Sec
  • Ian Ozsvald
  • Ilian Iliev
  • Import Python
  • Intellimath blog
  • Ionel Cristian Maries
  • Iraj Jelodari
  • IronPython-URLs
  • Ishan Chattopadhyaya
  • Israel Fruchter
  • Itamar Turner Trauring
  • Ivan Krstic
  • Jack Diederich
  • Jacob Perkins
  • Jahongir Rahmonov
  • Jaime Buelta
  • Jamal Moir
  • James Bennett
  • James Mills
  • Jan-Carel Brand
  • Janusworx
  • Jarrod Millman
  • Jason Meyers
  • Jason Webb
  • Jaysinh Shukla
  • Jean-Louis Fuchs
  • Jean-Paul Calderone
  • Jeethu Rao
  • Jeff Bisbee
  • Jeff Bradberry
  • Jeff Hinrichs
  • Jeff Knupp
  • Jeff Shell
  • Jeremy Epstein
  • Jeremy Hylton
  • Jim Fulton
  • Jim Hughes
  • Joe Abbate
  • Joe Pitz
  • Johan Dahlin
  • John Anderson
  • John Burns
  • John Cook
  • John Jacobsen
  • John Paulett
  • Jon Parise
  • Jonathan Dobson
  • Jonathan Ellis
  • Jonathan Harrington
  • Jonathan Hartley
  • Jonathan LaCour
  • Jonathan Street
  • Jorge Niedbalski
  • Jorge Puente Sarrín
  • Jorgen Schäfer
  • Juan Manuel Contreras
  • Juan Rodríguez Monti
  • Juho Vepsäläinen
  • Julien Danjou
  • Julien Palard
  • Julien Tayon
  • Juri Pakaste
  • Just a little Python
  • Justin Mayer
  • Kai Lautaportti
  • Karim Elghamrawy
  • Kate Editor
  • Kay Hayen
  • Kay Schluehr
  • Kelly Yancey
  • Kenneth Reitz
  • Kevin Dahlhausen
  • Kevin Dangoor
  • Kiwi PyCon
  • Kogan Dev
  • Konrad Delong
  • Koodaamo
  • Kracekumar Ramaraju
  • Kristján Valur Jónsson
  • Kriti Godey
  • Kulbir Saini
  • Kumar McMillan
  • Kushal Das
  • Károly Nagy
  • Laurence Rowe
  • Laurent Luce
  • Laurent Szyster
  • Leigh Honeywell
  • Lennart Regebro
  • Leonhard Vogt
  • Lightning Fast Shop
  • Lintel Technologies
  • Lion Kimbro
  • Lionel Tan
  • Logilab
  • Low Kian Seong
  • Luca Botti
  • Ludovic Gasc
  • Ludvig Ericson
  • Luke Macken
  • Luke Plant
  • Maciej Fijalkowsk
  • Made With Mu
  • Mahmoud Hashemi
  • Malthe Borch
  • Manuel de la Pena Saenz
  • Marc Garcia
  • Marc Kerins
  • Marc Richter
  • Marc-André Lemburg
  • Marcos Dione
  • Marius Gedminas
  • Mark Dufour
  • Mark McLoughlin
  • Mark McMahon
  • Mark Mruss
  • Mark Ramm
  • Marko Samastur
  • Martijn Faassen
  • Martijn Pieters
  • Martin Fitzpatrick
  • Mathieu Virbel
  • Mats Kindahl
  • Matt Harrison
  • Matt Layman
  • Matt Reid
  • Matt Wilkes
  • Matthew Rocklin
  • Matthew Rollings
  • Matthew Wilson
  • Mattias Brändström
  • Mauveweb
  • Max Ischenko
  • Michael Bayer
  • Michael Becker
  • Michael Droettboom
  • Michael Foord
  • Michael Hudson
  • Michael Nelson
  • Michael Sparks
  • Michal Kwiatkowski
  • Michał Bultrowicz
  • Michele Simionato
  • Michy Alice
  • Mike C. Fletcher
  • Mike Driscoll
  • Mike Müller
  • Mikeal Rogers
  • Mikhail Korobov
  • Mikko Ohtamaa
  • Mitch Chapman
  • Mitchell Garnaat
  • Mitya Sirenef
  • Montreal Python User Group
  • Morten W Petersen
  • Moshe Zadka
  • Moya Project
  • Mozilla Web Development
  • Muharem Hrnjadovic
  • Mycli
  • Nadav Samet
  • Nadia Alramli
  • Naomi Ceder
  • Natan Zabkar
  • Nathan Lemoine
  • Neckbeard Republic
  • Ned Batchelder
  • Neil Schemenauer
  • Ng Pheng Siong
  • Nicholas Amorim
  • Nick Coghlan
  • Nick Craig-Wood
  • Nick Efford
  • Nick Janetakis
  • Nicola Iarocci
  • Nicolas Dumazet
  • Nicolas Paris
  • Nigel Babu
  • Nikola
  • Nikolaos Diamantis
  • Not Invented Here
  • NumFOCUS
  • Obey the Testing Goat
  • Ofosos
  • Oliver Andrich
  • Omaha Python Users Group
  • Ondřej Čertík
  • OpenHatch Python posts
  • Paolo Melchiorre
  • Pathwright
  • Patrice Neff
  • Patricio Paez
  • Patrick Kennedy
  • Patrick Stinson
  • Paul Bissex
  • Paul Everitt
  • Paul Harrison
  • Paul Redman
  • Paweł Fertyk
  • PayPal Engineering Blog
  • Pedro Lima
  • Petecode
  • Peter Bengtsson
  • Peter Eisentraut
  • Peter Fankhänel
  • Peter Harkins
  • Peter Hoffmann
  • Phil Hassey
  • Philip Jenvey
  • Philip Semanchuk
  • Philipp von Weitershausen
  • Philippe Normand
  • Phillip J. Eby
  • Pierre Carbonnelle
  • Podcast.__init__
  • Polyglot.Ninja()
  • Possbility and Probability
  • Pradeep Gowda
  • Pranav Pandey
  • Praveen Gollakota
  • Programando Ciência
  • Programiz
  • Programming Ideas With Jake
  • Przemysław Kołodziejczyk
  • PyBites
  • PyCarolinas
  • PyCharm
  • PyCoder’s Weekly
  • PyCon
  • PyCon Australia
  • PyCon Pune
  • PyCon.DE & PyData Karlsruhe
  • PyPy Development
  • PyTennessee
  • PyTexas
  • Pypix
  • Python 4 Kids
  • Python 411 Podcast
  • Python Advocacy
  • Python Anywhere
  • Python Bytes
  • Python Celery - Weekly Celery Tutorials and How-tos
  • Python Data
  • Python Diary
  • Python Does What?!
  • Python Engineering at Microsoft
  • Python Insider
  • Python Open Mike
  • Python Piedmont Triad User Group
  • Python Software Foundation
  • Python Sweetness
  • Python Testing Cookbook
  • Python User Groups
  • Python on Karan
  • Python with Myo
  • Python(x,y) News
  • PythonClub - A Brazilian collaborative blog about Python
  • PythonDebugging.com
  • PythonThreads
  • Pythonicity
  • Pythonology
  • Péter Szabó
  • Péter Zsoldos
  • R David Murray
  • RMOTR
  • Ralph Bean
  • Ralph Heinkel
  • Ram Rachum
  • Randell Benavidez
  • Randle Taylor
  • Randy Zwitch
  • Raymond Hettinger
  • Read the Docs
  • Real Python
  • Red Hat Developers
  • Reinout van Rees
  • Rene Dudfield
  • Reuven Lerner
  • Richard Gomes
  • Richard Jones
  • Richard Tew
  • Richard Wall
  • Rickard Lindberg
  • Rob Galanakis
  • Rob Golding
  • Rob Miller
  • Robert Brewer
  • Robert Collins
  • Robert Zaremba
  • Roberto Alsina
  • Robin Parmar
  • Robin Wilson
  • Roche Compaan
  • Rodrigo Araúj
  • Roman Imankulov
  • RoseHosting Blog
  • Ruslan Spivak
  • Ryan Cox
  • S. Lott
  • S. R. Krishnan
  • SDJournal
  • SPE Weblog
  • Salim Fadhley
  • Salman Haq
  • Samuel Sutch
  • Sandipan Dey
  • Sandro Tosi
  • Sayan Chowdhury
  • Scripting the web with Python
  • Sean McGrath
  • Selena Deckelmann
  • Semaphore Community
  • Senthil Kumaran
  • Server Density
  • Shannon -jj Behrens
  • ShiningPanda
  • Shopkick Tech Blog
  • Shyama Sankar Vellore
  • Simeon Franklin
  • Simeon Visser
  • Simon
  • Simon Brunning
  • Simon Wittber
  • Simple is Better Than Complex
  • SoftFormance
  • Speed Matters
  • Spike ekipS
  • Spyder IDE
  • Stack Abuse
  • Stacks
  • Stanislas Morbieu
  • Starzel.de
  • Stefan Behnel
  • Stefan Petrea
  • Stefan Scherfke
  • Stein Magnus Jodal
  • Stephen Ferg
  • Steve Dower
  • Steve Holden
  • Steven Klass
  • Stijn Debrouwere
  • Stories in My Pocket
  • Stuart Gordon Reid
  • Stéphane Wirtel
  • Sumana Harihareswara - Cogito, Ergo Sumana
  • Sumith - Blog about SymPy/Python
  • Swaroop C H
  • Swisscom ICT
  • Sylvain Hellegouarch
  • Talk Python to Me
  • Tarek Ziade
  • Taylor Edmiston
  • Techiediaries - Django
  • Ted Leung
  • Tennessee Leeuwenburg
  • Terri Oda
  • Terry Jones
  • Terry Peppers
  • Test and Code
  • The Artificial Intelligence Cookbook
  • The Data Scientist
  • The Digital Cat
  • The No Title® Tech Blog
  • The Occasional Occurrence
  • The Parcon Blog
  • The Python Papers
  • The Three of Wands
  • TheAutomatic.net
  • Thibauld Nion
  • Thomas Guest
  • Thomas Vander Stichele
  • Thomi Richards
  • Tiago Montes
  • Tibo Beijen
  • Tim Gilbert
  • Tim Golden
  • Tim Knapp
  • Tim Lesher
  • Tim van der Linden
  • Tobias Ivarsson
  • Toes to the Edge
  • Tom Christie
  • Tomasz Ducin
  • Tomasz Früboes
  • Tomaž Muraus
  • Tomer Filiba
  • Tony Breyal
  • Torsten Engelbrecht
  • Toshio Kuratomi
  • Travis Oliphant
  • Trey Hunner
  • Troy Melhase
  • Tryton News
  • Turnkey Linux
  • Twisted Matrix Labs
  • Two Bit Arcade
  • Uche Ogbuji
  • United Coders
  • V.S. Babu
  • Varun Nischal
  • Vasudev Ram
  • Vinay Sajip
  • Vinay Sajip (Logging)
  • Vinta Software
  • Virgil Dupras
  • Vladimir Iakolev
  • Vladimir Perić
  • Wallaroo Labs
  • Wallix
  • Wayne Witzel
  • Weekly Python Chat
  • Weekly Python StackOverflow Report
  • Wes Mason
  • Wesley Chun
  • Wichert Akkerman
  • Will Kahn-Greene
  • Will McGugan
  • Will Pierce
  • William Minchin
  • William Reade
  • William Thompson
  • Wingware Blog
  • Wingware News
  • Wraithan
  • Yaco
  • Yaniv Aknin
  • Yann Larrivée
  • Yasoob Khalid
  • Yuval Greenfield
  • Zaki Akhmad
  • Zato Blog
  • Zero-with-Dot (Oleg Żero)
  • ZeroDB
  • bottlepy-dev
  • codeboje
  • codingdirectional
  • eGenix.com
  • eRambler
  • gamingdirectional
  • hypothesis.works articles
  • kdev-python
  • leftmouseclickin
  • meejah.ca
  • nl-project
  • pgcli
  • py.CheckIO
  • pythonwise
  • qutebrowser development blog
  • saaj/recollection
  • tryexceptpass
  • Éric Araujo
  • Łukasz Langa
  • بايثون العربي
  • To request addition or removal:
    Open an issue on github
    or e-mail planet at python.org
    (note, responses can take up to a few days)