Just moozing

Before you can check your notes, you must make them…

Python sandwiches

with one comment

python-logoPython can be used to make large and small programs. I refer to the small ones as scripts, and they are normally characterized by the fact that they take some well-defined data, processes it and outputs something new. This blog post is about how to think about these sandwich programs.

This is a very distinct category of programs that we use all the time, from bulk image processing, copying csv data in to a database or visualize data.

 

Designing

The design process is simple

  1. How does the input data look
  2. What do you want output to be?
  3. Find appropriate python modules
  4. Code it
  5. Match the output with your example

Most of these scripts will have very poor error handling and testing, since they do something very well defined, are used only by you and if they fail, you just fix the data/program and run it again.

 

Fundamental I/O data to use

Python is very useful for a lot of different data type and sources.

Common structured data formats:

There are a lot of data types, and depending on the application and the genre – different files will be common. Less simple, but still useful are HTML, XML, markdown, png/jpeg/other images and videos.

Besides the common files, python works nicely with databases. All the mayor databases have some sort of python bindings. For purpose of scripting, sqlite is really useful – since it is a database-in-a-file it very copy and paste friendly and having data in an sqlite database enables easy python reading/writing as well as many other languages.

 

Finding python modules

I find that googling “python <dataformat>” yields good results. Since python includes a lot of packages out-of-the-box, there are some recurring sites

It is well worth your time to look for established modules which handles the functionality you need. If you are creating images, processing an IMAP mailbox or scraping web pages, somebody has already done most of the hard work for you.

 

Always make an example first

I find that most issues arise, when the goal is unclear. So creating a good example input and output, will help keep focused.

Are you reading real-time data from a device, outputting to a production server or in a similar situation where it is not possible to make a good example, I use mocks. This is a concept from doing e.g. unittesting, where the actual input or output is faked.

If I want to mock the input, I make sure that my script design includes an instantiation of an object that returns my input data. That way I can easily create an alternate object, that just return some static data. That is usually sufficient when coding the program.

 

Yes, it is simple.

I have create a github project with code snippets to convince you that python is very good at these things.

Advertisements

Written by moozing

January 10, 2016 at 12:00

Posted in Tech

Tagged with

One Response

Subscribe to comments with RSS.

  1. […] my last post, I wrote about python sandwiches. Naturally I felt inspired, and made one myself. This one takes the zim tasks list and converts it […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: