When I post, I often get messages from people curious about whether they can use a package or model in production. For me, it’s been fun and enlightening reading Chip's book. This series of notes will hopefully be useful to anyone curious about how ML systems are built and deployed. Of course, if you’d like more in-depth information, you can buy the book here. I won’t be sharing everything I read but will try to write about the things I think are particularly helpful. In this post I’ll share some of the things from my notes on chapter 1!

What is a Machine Learning System

A misconception many people have about Machine Learning Systems is that they are just ML algorithms e.g logistic regression or different types of neural networks. In fact Machine Learning Systems include:

• Business requirements
• An interface for ML system users and developers
• deployment, monitoring and updating of logics
• Feature engineering
• ML algorithms
• Evaluation
• The data stack
• Infrastructure that enables the delivery of that logic

Here’s a visual of the different components of an ML system that the book covers

Huyen, Chip. Designing Machine Learning Systems (Fig, 1.1, p. 21). O'Reilly Media. Kindle Edition.

Regardless of which ML algorithm you use, this framework should still work!

When to use Machine Learning

I found this definition of machine learning from the book particularly helpful to keep in mind:

'Machine learning is an approach to (1) learn (2) complex patterns from (3) existing data and use these patterns to make (4) predictions on (5) unseen data.'

Huyen, Chip. Designing Machine Learning Systems (p. 22). O'Reilly Media. Kindle Edition.

Machine learning cannot solve all our problems and ML is often not the most optimal solution for solving a problem. From the definition, ML systems need to have the capacity to learn from data that has complex patterns. For example ML could be good for predicting the price of an Airbnb based on square footage, number of rooms, neighborhood etc. It is not appropriate to sort a list of Airbnb’s into states if the list comes with zip codes since this pattern is simple. It’s also important to note that existing data needs to be available or be possible to collect. ML models solve problems that require predictive answers. To predict is to estimate a value in the future. You can reframe most problems predictively. For example not asking, ‘who won the game’, but ‘who will win the game.’

Here are 4 characteristics of a problem where ML will shine:

• It’s repetitive
• The cost of wrong predictions is cheap
• It’s at scale
• The patterns are constantly changing

Here are 3 situations where you should not use ML:

• In an unethical way
• When there are simpler solutions that do the trick
• It’s not cost effective

The Most Common Ways Enterprises are Using ML

One of my favorite images from the first chapter was this diagram from Algorithmia’s 2020 state of enterprise machine learning survey. Huyen notes that even though the market for consumer applications is growing, the majority of ML use cases are still in the enterprise world.

ML applications in enterprises 'serve internal use cases (reducing costs, generating customer insights and intelligence, internal processing automation) and external use cases (improving customer experience, retaining customers, interacting with customers.)' Huyen, Chip. Designing Machine Learning Systems (p. 32). O'Reilly Media. Kindle Edition.

The figure below from 2020 shows us that ‘reducing costs’ was the main way cooperates used ML, followed by ‘generating customer insights/intelligence.’

Figure 1-3. 2020 state of enterprise machine learning. Source: Adapted from an image by Algorithmia. Huyen, Chip. Designing Machine Learning Systems (p. 33). O'Reilly Media. Kindle Edition.

The full report can be found here and the Algorithmia’s report on 2021 Enterprise ML Trends can be found here.

This is the first in a series of posts I’ll share here on my website. As I mentioned earlier, if you’d like more information on any of this you can buy the book here. Thanks for reading and follow my page for more!

Ibis provides a more Pythonic way of interacting with multiple database engines. In my own adventures I’ve always encountered Ibis’s (the bird versions) perched on top of elephants. If you’ve never seen an elephant in real life I can confirm that they are huge, complex creatures. Here’s a picture of me standing next to some for scale ;)

The image of a small bird sitting on top of a large elephant serves as a metaphor for how ibis provides a less complex, more performant way for users to interact with multiple big data engines. In fact the bigger and more complex your data the more of an argument there is to use Ibis. SQL can be quite difficult to maintain when your queries are very complex. With ibis there is no need to completely rewrite your code whether you’re scaling up or down. You can switch out your back-end if you need to and keep working in the same context but with a more powerful engine. This means your workflow is streamlined, less error prone and your cognitive load is reduced.

At the moment, Ibis supports quite a few backends including:

Traditional DBMSs: PostgreSQL, MySQL, SQLite Analytical DBMSs: OmniSciDB, ClickHouse, Datafusion Distributed DBMSs: Impala, PySpark, BigQuery In memory analytics: pandas, Dask

Whether you’re wanting to interact with SQL databases or wanting to use distributed DBMSs, Ibis lets you do this in Python. In this post I’ll be using an SQLite backend but if you’d like to try any of the others checkout the Ibis backends page for examples. Ibis currently supports over 12 backends, some of which compile to SQL and some of which are directly executed.

For Python programmers Ibis offers a way to write SQL in Python that allows for unit-testing, composability, and abstraction over specific query engines (e.g.BigQuery)! You can carry out joins, filters and other operations on your data in a familiar, Pandas-like syntax. Overall, using Ibis simplifies your workflows, makes you more productive, and keeps your code readable.

Let’s use the Civic Art Collection dataset I mentioned earlier and see what Ibis can do!

Installation

You can install Ibis with Pip, Conda or Mamba.

For the sake of security, I’d recommend setting up a virtual environment. You can copy and paste these commands into your terminal to install ibis with either Pip or Conda.

Pip:

In this tutorial we’ll be using SQLite as our backend. If you use pip to install Ibis you’ll need to run pip install 'ibis-framework[sqlite]' instead of the usual pip install ibis-framework for sqlite to work. If you’d prefer to use another backend look for the specific commands in the Ibis documentation.

Mac:

python3 -m venv ibisdev
source ibisdev/bin/activate
pip install 'ibis-framework[sqlite]'
pip install pysqlite

Windows:

python3 -m venv ibisdev
ibisdev\Scripts\activate.bat
pip install 'ibis-framework[sqlite]'
pip install pysqlite

Conda

conda create -n ibisdev python=3.8
conda activate ibisdev
conda install -c conda-forge ibis-framework
conda install -c anaconda sqlite

Creating a database with SQLite

If you’d like to follow along with me, the next thing we’ll do is create a database and a table containing data about art around San Francisco from DataSF. The data is open source so you can download the csv file directly from the site. Ibis doesn’t support loading csv files into an SQL table yet, so we’ll follow these commands to do so in the command line with SQLite

Create a folder called civic_art

mkdir civic_art
cd civic_art

To make things easier in the next steps, I moved the csv file from my downloads folder into the civic_art folder we just created. While we’re in the civic_art folder we create a new database with the following command.

Create a database named civicArt.db

sqlite3 civicArt.db

This command should also open SQLite in your command line. Here’s a picture of what this looks like for me (I used .databases to double check that the database was correctly created.)

Great! The database has successfully been created! SQLite opens up already in the civicArt database so the next thing we’ll do is create a table and load the data from the csv file into it.

Create a table in the civicArt.db database called civicArtTable

.mode csv civicArtTable (names the table) 
.import Civic_Art_Collection.csv civicArtTable (import the data into the table)

Check that the table has been created as expected

.tables   (this lets you check the table has been correctly created) 
.schema civicArtTable (describes the table)

If your table has been created as expected you can now exit SQLite by typing in .quit. This should take you back to your command line. We’re now ready to start exploring our data with ibis!

Connecting to the database

One of the advantages of ibis is that it lets you work with tools you’re most comfortable with. I enjoy writing Python code in IPython, but you can use a jupyter notebook or any Python shell you’d like. Whatever you choose we'll use the same commands to connect to the civicArt database we created earlier.

Connecting to our database (we’ll be using Ibis’ interactive mode for lazy evaluation)

import ibis
ibis.options.interactive = True
db = ibis.sqlite.connect("civicArt.db")

Something to note here is that pandas (read_sql) loads data into memory and performs the computations itself. Ibis won't load the data or perform any computation. Instead it leaves the data in the database defined by the connection, and will ask the backend to perform the computations there. This means you can execute at the speed of your backend, not your local computer. A good way to think about this is to go back to our elephant metaphor. If you need some heavy lifting done, it’s probably a good idea to let an elephant do it as compared to an ibis. Our elegant ibis can pass on the heavier work to the larger database and watch as the work is done much faster than it could have done itself.

If everything went well you should be connected to the database. We’ll take a look at our data next!

Viewing Table Details

Let's use Ibis to take a look at the tables in the civicArt database

List out all the tables in the database

 db.list_tables()

Assign the table to a more readable name and list the columns in it

art = db.table("civicArtTable")
art.columns

This is what I’m seeing in IPython after calling these commands

Wrangling Data With Ibis

Let's now use Ibis to carry out some common SQL commands on our dataset and find out some useful information about art in San Francisco in the process.

Querying

Anything you can write in a SELECT statement you can write in Ibis. Let's test this out! I’ll use the following code to find out which artists have art currently displayed in the city and what the title of their pieces are.

Selecting columns from a table

art["artist", "display_title"]

Great! If you’ve used Pandas before this should feel similar to getting columns from a Pandas Dataframe, but remember the database is doing all the work for us, so this is more efficient!

I’m also curious about how many artists total have their work on display in case an artist has more than one piece exhibited. To do this let's get distinct artists and then count them

Distinct and Count

distinct_artists = art["artist"].distinct()
distinct_artists.count()

We can already see the length of the list, but just to show you the count method, we confirm that the number of artists that have one or more pieces on display in San Francisco is 386!

Filtering Data

Next let's pick an artist and figure out where exactly all their art is located. Adriane Colburns’ display title, Geological Ghost caught my eye so let's choose them!

I use the following commands to do this

adrianes_art = art.filter(art["artist"] == 'Colburn, Adriane')
adrianes_art

So it turns out Adriane has two pieces on display, one at 4 Guy Place and the other at Dagget Street & 16th Street. This is great, we already have some places we can add to our tourist itinerary!

Groupby

I don’t usually stay more than one or two days in a city after a conference, so it might be nice to know which locations have the most art on display. To figure this out we’ll use the following groupby expressions to get the information we need.

Use groupy and sort_by to get locations in San Francisco with the most art!

art_loc = art.groupby("street_address_or_intersection").count('display_title')
most_art = art_loc.sort_by(ibis.desc(art_loc.display_title))

Great, the first stop would be 1001 Potrero Avenue which has 59 titles on display!!

Out of curiosity, I searched 1001 Potrero Avenue to learn more about what I expected would be an art gallery. Surprisingly it turned out to be the Zuckerberg San Francisco General Hospital. They’ve apparently invested a lot in art and have an incredible collection. The thing with data is that it can surprise you! The next best option is Coit tower which has its address at 1 Telegraph Hill Blvd. It is not a hospital and looks like a great place to explore!

credit: Janice Heckert

Joins

Excellent. At this point we know that the next time we’re in San Francisco for a tech conference we have some great options to check out. I only found out after writing my code so let's imagine 1001 Potrero Avenue is not a hospital and use it as our first choice! Now we know where to go. What's next?

Well, oftentimes art galleries are filled with hipsters that will likely ask you about the artists on display. The good thing is that we can prepare for this in advance! Lets use an inner join to help us find which artists have the most art showing so we can look them up and feel cool by having a good response ;)

Use inner_join to find the most popular artists

artist_location = art["street_address_or_intersection", "artist"]
artists_at_portrero = artist_location.filter(artist_location["street_address_or_intersection"] == "1001 Portrero Avenue").distinct()

*ibis version 2.1.1 sometimes encounters an issue with views note being materialized and then materialize not resolving column names on a join operation. The following code fragment is just for version 2.1.1 and uses the mutate() function (first line) to change one of the column headers and materialize() to ensure that the created view can be displayed. This should be fixed in version 3.0.0.

artist_number_of_displays = art.mutate(c_artist=art['artist']).groupby("c_artist").count("display_title")

most_popular_artist = artists_at_portrero.inner_join(
    artist_number_of_displays,
    predicates=artists_at_portrero["artist"] == artist_number_of_displays["c_artist"]
    ).materialize()

Fantastic! We now know Mary (McChesney) Fuller has the most art on display at 1001 Potrero Avenue and can search online to find out more! In my opinion, we’re as ready as we’ll ever be to explore the art scene in SF!

We learned a lot of information in a relatively short amount of time. If you’re used to writing Python code this probably feels more efficient than writing out raw SQL strings. It also means if you started your workflow in IPython or a Jupyter Notebook you can stay there. This should feel smoother and hopefully lead to increased productivity. Using tools that help you be more efficient means you have more time for creativity and exploration in your work! I wish this for you! Thanks for reading and hopefully I’ll bump into you watching elephants or in an art gallery somewhere around the world!

We've also made a jupyter notebook that has all of the commands in this post! You can find it here

Look for the docs📜

The first step in contributing to a library you're new to is to look at the documentation. Most OS projects have a contributors guide. There's a new and improved arrow guide thats just been published. You can keep following along with my post, but I'd definitely recommend checking out the guide if you run into any errors I don't address. You can check it out here.

Choose an issue to work on🛠️

If a project needs help they'll write an 'issue' for it. In most cases issues are on Github. For @ApacheArrow issues are on Jira. Even If you aren't ready to contribute just yet, looking at a projects issue list lets you know what you can prepare for once you are ready.

This was the issue my 1st issue.

Some things to note about it:

-It's a python issue (there were also #r, #cpp, #js issues) -The goodfirstissue label is visible -It says closed because I took the screenshot after but you want to look for open issues

Set up your dev environment🧑🏿‍🔬

Even if you're not exactly sure how you'll solve an issue, the next step should be to set up your dev environment. I usually only think of a solution once I've experimented with the code. For PyArrow setting up looked like forking and cloning the apache arrow repository, setting up a python virtual environment, building the arrow libraries.

Forking the Arrow Github repository

You can find the arrow repo is here. You can find a tutorial by Github on forking here

Cloning the Arrow Github repository 🐑

Cloning a repository creates 2 remote branches origin and upstream in a directory named arrow. In order to do this I used the commands seen in the image below

At this point you should be in a folder named 'arrow'. This would also be a good time to create a Python virtual environment.

Building arrow c++🧱:

In order for you to run arrow on your local machine you need to build both its c++ and python libraries. You can find documentation about the commands you need to do this using different OS' here.

I'm on Mac so I used the commands specified for it. I also used the projects cmake presets to get started quickly, though if you're planning to contribute more frequently building without presets might be best. I found that building the libraries tends to be the trickiest part. If you run into any issues feel free to reach out to me on Twitter or Github. There's also an apache arrow mailing list which you can send questions to as well. Here's the list of commands I used:

Building pyarrow🧱: The instructions are here👇🏿

The code to needed to build PyArrow will vary based on your OS. Here's a page that lists the options. I used the commands shown in the image below (as a reminder I'm on a MAC)

Figuring out how to solve the issue🧐

When I'm trying to solve an issue, my first goal is to understand how the function in question works. To do this I'll

-open up IPython or a jupyter notebook, which is where I'm most comfortable working. -I'll then search Github or my editor to find where the function -I'll call the function and make sure I understand what it does. This also helps me understand the problem -Then I'll try solutions in my branch, read errors, and ask questions about the issue if needed. Feel free to comment on the issue itself to get answers about things you aren't sure about. Here's a picture of a discussion I had about the issue I was working on

Running Tests and Linters👩🏿‍🏫

PyArrow uses pytest for unit tests in Python. You'll need to cd into the python/pyarrow folder.The project also uses archery for linting purposes! Different OS projects use different linters so check the docs! Here's the code for all of this.

Making a Pull Request✨

If you have a working solution try not to overthink! Sharing code means other people can help! I like to check which files have changed to make sure I didn't change any by mistake. I'll add and commit my code with a clear message and then push the changes. Here's the code I used for this

Code Reviews✅

Once you've pushed your code, give your PR a descriptive name. Maintainers will then comment on your code. Don't be afraid to receive feedback, even though this can be scary. If you get feedback make changes in your local branch and then and push them to your PR. Here's a look at what my PR looked like once I pushed it and started getting reviews.

Here's a link to the issue and PR , incase its helpful:

Issue: https://issues.apache.org/jira/browse/ARROW-14242

PR: https://github.com/apache/arrow/pull/11873

That's it🎉✨ If you're thinking of contributing to OS (not just @ApacheArrow ) hopefully you found this helpful! Let me know on twitter if you end up creating a PR of your own.

Setting Up

To start with we’ll use some of the code from the last post to create some background scenery. Here’s the image I have in my browser. I’ve also included the sketch file code for this initial scene. If you run the code and don’t see anything in your browser, head back to my last post to see why you might be seeing a blank screen, or check your console for any errors.

var floorPos_y;
var gameChar_x;
var gameChar_y;

function setup()
{
    createCanvas(1024, 695);
    floorPos_y = height * 3/4;
    gameChar_x = 300;
    gameChar_y = floorPos_y;


}

function draw()
{
    background(255,215,0); // fill the sky with gold

    // draw some yellowish ground
    noStroke();
    fill(240,230,140);
    rect(0, floorPos_y, width, height/4); 

    // Draw clouds.

    noStroke();
    fill(255);

    noStroke();
    fill(255, 80);
    rect(320, 122 - 80 / 3, 80, 40, 100);
    rect(320, 122, 160, 40, 100);

    // Draw trees.

    stroke(0);
        //trunk
        fill(205,133,63);
        rect(730,floorPos_y * 6/10 -50,20, 260);

        //leaves

        fill(0,128,0);
        noStroke();
        ellipse(710,(floorPos_y * 6.7/10) -70,90,90);

        ellipse(760,(floorPos_y * 6.7/10) -70,100,120);

    // Draw canyons.

    noStroke();
    fill(color('#666699'));
    rect(10, 520, 100, 186);
    fill(color('#cce6ff'));
    rect(10 + 20, 550, 100 - 40, 190);
    fill(255,215,0);
    rect(10 + 20, 482, 100 - 40, 68);
}

Variables

Something different you might notice right away is the first 3 lines of code in this file;

var floorPos_y;

var gameChar_x;

var gameChar_y;

In the previous post, I didn’t touch on variables so I’ll do so briefly now (if you have experience with variables skip to the next paragraph.) The last time we used p5js functions like ellipse() to draw shapes on the screen. Each function took a set of ‘arguments’, for example, the ellipse() function takes a value for the shapes x positions, y position and its size. We used fixed values for these arguments because the shapes were staying in one place, and not changing size. However, when we want to add animations and movement to an object (like our game character), using fixed values won’t be sufficient. This is where variables come in to save the day!

Variables are placeholders that can be substituted by new values or adjusted once the program starts to run. In our code floorPos_y, gameChar_x and gameChar_y are variables. You can create or ‘declare’ a variable by using the ‘var’ keyword at the top of the program. We then give it an initial placeholder value. In this case, in the setup function, we have set gameChar_x to 300, floorPos_y to ¾ x height and gameChar_y to the same value as floorPos_y. Throughout our program, we will assign different values to these variables, and they will take on that new value at that moment. We’ll become more comfortable with this concept as we develop our game character, so let’s start that right away!

The floorPos_y variable is the y value for the top of the rectangle representing our floor. The gameChar_x and gameChar_y variables will represent where our game character is located. Note that I’ve given each of them an initial value in the setup function. Let's start by drawing an ellipse on our ground. Add the ellipse() function at the bottom of the draw() function and for its arguments, make the first argument gameChar_x and the second gameChar_y. Then, use 30 and 30, for the third and fourth arguments to make the circle 5x5 pixels (if you need a refresher on drawing shapes in p5js visit this link.) Finally, put the fill() function above the ellipse() function and make it any color you want. I’ll go with a pinkish color for now. In the live preview window, you should now see the ellipse sitting on the ground. You should also note that the program reads your code from the top to the bottom, this is the ‘flow’ of the program. The code at the top of a function is drawn first, so is placed in the background and the code at the bottom is drawn last and so is placed in the foreground. This is why the code for your ellipse must be at the bottom of the draw() function. If you aren’t seeing it on the screen it may be because you have it in the wrong order in your draw() function.

keyPressed and keyReleased Functions

I’m now going to introduce you to two new functions, keyPressed and keyReleased. Similarly to the setup and draw functions, these are functions that p5js gives us. If you have time, head over to the links to read more about what these functions do. In summary, the keyPressed function tells the program what code to run when a key on your keyboard is pressed. The keyReleased function tells it what to do when a key has been released after being pressed. Add these two functions at the end of your program, after the draw function as shown below.

function keyPressed()
{

}

function keyReleased()
{

}

Conditional Statements

Next, we’re going to add what are called 'conditional statements' to the keyPressed and keyReleased functions. Conditional statements are like questions. They allow a program to decide to take one action if the answer to a question is true or to do another action if the answer to the question is false. We’ll start to write one now in the keyPressed function.

We want to tell the program what to do when the left or right arrow keys are pressed. To do this we will write two ‘if’ statements. When writing an 'if' statement start by writing the word if, followed by closed brackets and some curly braces. Like this
if(){}
The closed brackets take what we call a 'condition' for its argument. The condition determines whether or not the code in the curly braces is run. For our first conditional statement, we will be using the 'keyCode' of the left arrow as our condition. You can read more about keycodes here. Take a look at the code below.

function keyPressed()
{

    if(keyCode == LEFT_ARROW)
    {

    }

    else if(keyCode == RIGHT_ARROW)
    {

    }

}

function keyReleased()
{

    if(keyCode == LEFT_ARROW)
    {

    }

    else if(keyCode == RIGHT_ARROW)
    {

    }

}

If the left arrow is pressed the LEFT_ARROW keyCode is recognized and the code in the curly braces runs. Therefore the first conditional statement tells the program what to do when the left arrow key is pressed. The second one tells the program what to do when the right arrow is pressed. The same thing goes for the statements in the keyReleased function.

So what happens when one of our keys is pressed or released? We’ll be placing some variables in these conditional statements that will determine what will be drawn to the screen. Let's start by creating two new variables.

At the very top of your program type in var isLeft and var isRight. In the setup() function type in isLeft = false and isRight = false. Setting these two variables to true or false makes them ‘boolean values.’ Boolean values are used quite often in programming and let us know if something is true or false. In` our case we’ll be using them to tell if the arrow being pressed and released is the left or right one. We initially set the values to false because we aren’t pressing either of those keys right now.

Inside the first ‘if’ statement, in the keyPressed function, add the isLeft variable and set it to true. Add the isRight variable to the second if statement. Do the same for the keyReleased functions but set the values to false. Your code should look similar to this (I've left out the code in the draw() function, and I've also added a new console.log() function, which we'll discuss next.)

var floorPos_y;
var gameChar_x;
var gameChar_y;
var isLeft; 
var isRight; 

function setup()
{
    createCanvas(1024, 695);
    floorPos_y = height * 3/4;
    gameChar_x = 300;
    gameChar_y = floorPos_y -13;
    isLeft = false; 
    isRight = false; 
}

function draw()
{ 
   ...
} 

function keyPressed()
{
    if(keyCode == LEFT_ARROW)
    {
        console.log('left arrow');
        isLeft = true;
    }

    else if(keyCode == RIGHT_ARROW)
    {
        console.log('right arrow');
        isRight = true;
    }
}

function keyReleased()
{
    if(keyCode == LEFT_ARROW)
    {
        console.log('left arrow');
        isLeft = false;
    }

    else if(keyCode == RIGHT_ARROW)
    {
        console.log('right arrow');
        isRight = false;
    }
}

Debugging with Console.log()

We haven’t made any changes to the draw() function so we won’t see any changes on the screen just yet. However, Javascript has a built-in function that can help us ‘debug’ or check our code is working. The function is console.log(). You can see that I’ve added some console.log() statements in my code above. If the ‘if’ statement works correctly, when its condition is met, the statement in the console.log() function should appear in your developer console. For example, with the code I have when I press the left arrow key I should see ‘left arrow’ in my developer console. This process of checking to see if your program is running smoothly in the console is the foundation of ‘debugging’ in Javascript. Console.log() is a function you can use to do this, even for more complex projects. This is what my console looks like when my conditions are working correctly.

If all your keys are showing the correct console messages Hooray!!! Lets start to get our character moving!

Changing Variable Values

In the draw() function we will add two new ‘if’ statements. The first one will be if(isLeft == true). This conditional will tell our program what to draw to the screen if the isLeft variable is true. If you remember, we just made it so that this variable is set to true, only if the left arrow key is being pressed. It is set back to false when the left arrow key is released again. In this conditional statement, we will place the drawing code for the ellipse (our character), but we will now be focusing on changing the value of the character's x position on the screen.

At the beginning of the post, we set the value of gameChar_x to 300. When the left key is pressed, we want the character to move to the left. On the x-axis, this means that the value of x should decrease. To do this, after the code to draw your ellipse in the isLeft ‘if’ statement place the statement gameChar_x -= 5. This statement will subtract 5 from the value of gameChar_x for as long as the condition in the isLeft statement is true. This is the same concept we will use for the isRight conditional. We’ll also add the original ellipse code in an ‘else’ statement right at the bottom of the draw() function. This makes sure the ellipse is still showing when none of these keys is pressed. Take a look at my code to see how this is done.

 if(isLeft)
    {
        gameChar_x -= 5;
        fill(240,128,128);
        ellipse(gameChar_x, gameChar_y, 30, 30);
    }

    else if(isRight)
    {
        gameChar_x += 5;
        fill(240,128,128);
        ellipse(gameChar_x, gameChar_y, 30, 30);
    }
    else
    {
        fill(240,128,128);
        ellipse(gameChar_x, gameChar_y, 30, 30);
    }

Other movements in our program like jumping, use these same concepts. Try and figure out how to do this on your own. [HINT] Create an isJumping variable (boolean) and subtract from the gameChar_y variable. Consider using the AND operator in your isJumping conditional statement in the draw() function to make sure the character only jumps when it's on the floor. Read about operators here. Also, try and add a new ‘if’ statement that detects when your character is above the ground and when this condition is true subtracts from gameChar_y to make it fall back to the ground. I’ll leave a link to my GitHub below for you to have a look at my full code, but try as much as you can to work it out on your own :)

Alright! We’ve done quite a bit already. If you’d like to improve the way your character looks when it moves you can edit the code in the draw() function under a specific conditional statement. For example, if you’d like the ellipse to appear thinner as it moves left, edit the size argument of the ellipse() function under the isLeft conditional. In my case, the ellipse code changed to this ellipse(gameChar_x, gameChar_y, 15, 30); and when I press the left arrow and the ellipse moves left, I see the following on my screen.

If you can make these changes, you can also start to edit your character so that it is not just an ellipse. Add other shapes to it, to make it as complex as you’d like, or add an image character. I’d advise you to start with the ellipse code in the ‘else’ conditional. This is the code that is displayed when your character is at rest. Remember to use the gameChar_x and gameChar_y variables as your arguments. This makes sure that all the shapes you’ve used adapt when we edit these variables. Start simple but as you grow in confidence you can make your character more complex. Here’s the character I started with. I only made my character more complex right at the end of creating my game. This is sometimes best as it can help you avoid confusion.

Ok! Well done if you managed to reach this point! You can already do so much with the knowledge you now have. You also learned the basics about some fundamental concepts in Javascript namely; variables and how to change them, booleans, conditional statements and debugging with console.log().

I think this is a good place to end this post for now. If you’d like to download the code that’s behind what we have so far, download it on my GitHub here.

I’d encourage you to play around with the code. Try and see if you can add more conditional statements, say for example to change what the character looks like when it's jumping and facing left. Try to add a statement that makes the character fall when it’s over the water. You should be able to do lots of things with conditional statements. If you struggle with anything let me know and I’ll address it in my upcoming post. Next time, we’ll create our functions and focus on making the character interact with its environment. This is useful for adding a points system to a game or adding or losing a life when you create villains. As usual, we’ll also learn some core concepts in Javascript while having fun with this game.

If you have any specific questions feel free to reach out to me! Send me an email here, or find me on Twitter. If you enjoyed this post share it with someone else!

Thanks for reading,
Marlene

Get Started with Ibis

import ibis
db = ibis.sqlite.connect('geography.db')
countries = db.table('countries')
countries.columns

Reading a parquet file

import pyarrow as pa,  pyarrow.parquet as pq
#read parquet file as pyarrow table 
table = pq.read_table("example.parquet")
print(table)

Write A Parquet File

import pyarrow as pa, pyarrow.parquet as pq, numpy as np
#create an array with 100 numbers from 0 to 99
arr = pa.array(np.arange(100))
#create a pyarrow table 
table = pa.Table.from_arrays([arr], names=["col1"])
pq.write_table(table, "example.parquet", compression=None)

Create a PR

git status
git diff
git add <filenames>
commit -m "messages>"
git push origin <branch-name>

Top 10 python data projects on Github

import requests
import pandas as pd
url = 'https://api.github.com/search/repositories?q=data+language:python&sort=stars&order=desc'
res = requests.get(url)
res_dict = res.json()
repos = res_dict['items']
repo_df = pd.DataFrame(repos)
repo_df.head(10)

Ewa Jodlowska, Executive Director, moderated a chat with two PSF Board Directors, Lorena Mesa and Marlene Mhangami. The group discussed the upcoming 2020/21 PSF Board Elections and what sort of tasks Board Directors have.

Recorded May 2020

Real Python interview

We are joined today by Marlene Mhangami. Marlene is a passionate Pythonista who is not only using tech to facilitate social change and empower Zimbabwean women but is also the chair of the very first PyCon Africa. Join me as we talk about her non-traditional start in tech, as well as her passion for using technology to create social change for good.

Recorded May 8th 2019

---

PyDev of the Week

This week we welcome Marlene Mhangami (@marlene_zw) as our PyDev of the Week! Marlene is the PyCon Africa (@pyconafrica) chair, the co-founder of @zimbopy and a director for the Python Software Foundation. Let’s spend some time getting to know her!

Recorded May 8th 2019

---

Recorded January 21st, 2019

Recorded October 1st, 2018

Setting Up

For this project, I’m using a language called Javascript, and along with this, a Javascript library called p5.js. A library is a collection of code written by someone else that we’re allowed to use in our programs. It makes the process of creating things a lot faster! For example, if you’re baking a cake, sometimes you can make it from scratch but sometimes, you can buy a pre-mix, and then add things from there to shorten the process, using a library is like that. P5.js is a great library that is designed to make graphical Javascript (JS) applications and animations. You need to have a code editor installed, where you’ll write and edit your code. If you’re just starting I’d highly recommend using Brackets, or Sublime Text. (Some popular code editors have a lot of buttons and lights, which I find can be intimidating when beginning.) Download and install Brackets here. Once you’ve done that the next step is to download the p5.js library. You should download the ‘Complete Library’ which you can find here. Ok, that’s a good starting point, let's dive into the code!

Getting Started

Open the downloaded p5.js folder by using the ‘Open Folder’ command under ‘File’ in your newly installed Brackets editor. This is what you should see if you click on the sketch.js file to the left. This is where we’ll be writing and editing all our code.

The sketch file contains two ‘functions,’ setup() and draw(). A function is a block of code that performs a specific task. You’ll learn what tasks these two functions carry out in just a bit. For now, if you’re using Brackets click the lightning symbol on the right-hand side of the screen. That opens a window in your browser (e.g. Chrome) and that gives you a live look at what you’ve created with your code. We haven’t written any code yet so that's why the screen is blank. At this point, I’m going to suggest you have a look at the p5.js get started page, and focus on trying to understand what the setup, draw and create functions do. You can also watch this tutorial which will help you understand how to draw shapes with P5.js. Once you feel confident drawing we’ll begin to add the background scenery to our game.

Basic Shapes

In the sketch file create a canvas for your game scene by adding the createCanvas() function inside the setup() function underneath where it says ‘put setup code here.’ (Just as a reminder a list of functions and what they do can be found here.) Make the canvas 1024 x 695 pixels, by typing in createCanvas(1024, 695). The numbers 1024 and 695, are what we call ‘arguments.’ The function needs you to put these arguments inside it to do its job. Functions can take different types and different numbers of arguments.

Next up choose the color of the sky by filling the canvas with any color you want! Add the background() function inside your setup() function as well. The background function takes RGB values, so making the background color of your canvas gold looks like this; background(255,215,0). Hooray! You have created the sky!!! Your screen should look something like this

Now we’ll add some clouds. Let's start by making our clouds with 3 ellipses, you can improve the look of your game a bit later on. The ellipse() function allows you to draw an oval shape, it takes the following arguments; an x-coordinate, a y-coordinate, a width, a height. The first two determine the ovals location, the second two determine its size. We add this ellipse function inside our draw() function, instead of in setup() function, we do this when we want to start adding drawings to the screen. Whoops, if you’ve added your ellipse to the code editor but aren’t seeing it appear in your canvas don’t panic. This is probably because of two things.

Firstly, we’ve forgotten to give our ellipse some color. You can do this by using the fill() function which takes RGB values for arguments and fills any shape below it with that color. To make your clouds white use fill(255). You can find other RGB color values here. You can also vary the transparency of your clouds by using the ‘alpha value’ in the fill() function. Read about that here. Play around with this to give your clouds different colors. If you’d like to make your clouds the same shape as mine from the game example I built you can use the rect() function to draw a rectangle and then add two extra arguments to round its corners. You can read about that here. Here’s what you should be looking at.

Another reason you could be getting a blank screen is that you’ve made a mistake in your code. That's fine, everyone does! A great way to check this is by checking our console. A console is a tool developers use to check what's happening with their program in the browser (e.g Chrome.) To access your console, at the top of your browser click the View menu option, scroll down to developer and then choose Javascript Console. Once it's open you’ll see several tabs there, ‘ Elements, Console and Sources’ hopefully are some you see. Click on the console tag and take a look if you have any messages appearing in red. These messages will let you know if you have any errors and what line the error is on. Here’s an example of a mistake I made.

I forgot to add the closing bracket in the setup function. The console told me the line to look for the error on (line 15 in the sketch.js file) and what was the issue. (If you’re not using Chrome, do a quick search online on where to find the Javascript Console using your browser.)

Hopefully, you’re back to see some colors and shapes in your browser. Next, we’ll create the ground by using the rect() function. This draws a rectangle to the screen (look at the p5js reference page again if you need a reminder on how this works.). Add this function to your draw function as well and make the x-coordinate be the start of the screen, and the height be about ⅓ of your height. For me a nice ground looks like rect(0, height/3, width, height/3), you can use the height and width commands in p5js. Ok, we should be looking good. Using this same logic add other things to your background scenery by drawing shapes. Use the triangle() function to draw a mountain, the rectangle and ellipse functions to draw some trees and so on. Have a look at my code here to get some ideas.

function setup()
{
    createCanvas(1024, 695);
    background(255,215,0); // fill with gold
    floorPos_y = height * 3/4;
}

    // Draw Mountains.

    fill(0);
    triangle(400, height * 3/4,
    400 + 230, height * 3/4,
    400 + (230 / 2), height * 3/4 - 320);

    // Draw trees.

    stroke(0);
        //trunk
        fill(205,133,63);
        rect(730,height * 3/4 * 6/10,20, 260);

        //leaves

        fill(0,128,0);
        noStroke();
        ellipse(710,(height * 3/4 * 6.7/10) -20,90,90);

        ellipse(760,(height * 3/4 * 6.7/10) -20,100,120);

    // Draw canyons.

    noStroke();
    fill(color('#666699'));
    rect(10, 520, 100, 186);
    fill(color('#cce6ff'));
    rect(10 + 20, 550, 100 - 40, 190);
    fill(255,215,0);
    rect(10 + 20, 482, 100 - 40, 68);

So far we’ve only been looking at basic shapes. We'll take a look at some more complex shapes and designs next!

Adding detail and background images

You might be feeling a bit frustrated by having to use basic shapes. The great thing is you don’t always have to. You can create custom shapes and designs. To do this you can play around with the vertex(), quadraticVertex, bezierVertex, beginContour() and endContour() functions! These can all be a bit trickier to master but can help improve the look and feel of your game. As usual, you can find them on the p5.js reference page, but if you’d like to read more in detail about this I’d recommend having a look at Rune Madsen's Programming Design Systems. If you’d prefer to watch a video tutorial have a look at this. The buildings in the background of my game, in the beginning, have lots of detailed designs on them. I used the vertex() and line() functions to create most of these. I added them to the code so you can have a look (I’m sure there must be better ways of creating these if you have ideas let me know!)

Creating your designs is fantastic and fun when you’re in a creative mood. Sometimes, though it means writing lots of code or coming up with a formula to make it easier. If you don’t feel like doing this, there is an alternative to use in case of emergencies. You can add real images to your game!

This can be a handy way to create background scenery in a hurry. You must think about the look and feel of your game, and try as much as possible to keep it consistent. Sometimes adding images can make the game look disorganized, but other times it can save you a lot of time and energy. To add images, you’ll have to first find some images you’d want to use. There are lots of sites that provide you with free creative images including creative commons. Make sure to only use images that have been shared with the public or that you’ve purchased from their creators.. You’ll also need to use the p5.js loadImage() and image() functions. You can see some examples and read specifics about how to use these functions here. In my game, I really wanted to add a traditional drum, mainly because they are awesome and something I felt would add to the game. A lot of traditional drums have super detailed designs and I thought that might be really tiring to write the code for haha, so I used an image. Here’s what that looks like, when I add it to our code so far.

Conclusion

If you’ve reached this point, you’ve already achieved quite a lot! You’re on your way to creating an incredible game. Play around with changing the background and making it your own. Maybe add images to the canvas or find other interesting functions to use from the reference page. I’ll be splitting this topic up into about 3 different parts. In the next post, we’ll learn how to create a unique game character and how to make it move around the screen. We’ll also start to look in detail at how to create our functions instead of only using the ones given to us by p5.js. This will help our game character interact with its surroundings. Exciting! Hope you enjoyed this first part and that you come back again for the second one!

The drum image I used is from verctorportal.com Have any specific questions? Send me an email here, or find me on Twitter.