Josh Winton

Adding Stats to The Doomsday Game

2021-12-09T00:00:00+00:00

Last time, we created a game for getting better at using the doomsday algorithm. In this post, we’ll look at ways to improve on this basic idea by keeping track of some relevant statistics. The statistics we’ll look at today are:

Games played
Games won

We’ll approach this task by writing these numbers to a JSON file. The JSON file format is easy to work with because it can represent a variety of data.

To implement this idea, we’ll write two functions: one that creates the results file if none exists and one that reads an existing results file and updates it with any new results.

Let’s start by looking at how to create this file:

def createResultsFile():
    results = {'games': 0, 'wins': 0}
    with open('results.json', 'w') as f:
        f.write(json.dumps(results))

While inside our program, we’ll represent our results as a dictionary. For now, we’ll just be keeping track of the total number of games and how many of those games were wins. Once the dictionary is created, we write it to a file in the JSON format. We’ll only call this function when the file doesn’t already exist, so it makes sense to start with zero games and zero wins.

Next, we can write the function that reads and updates the results file when it does exist:

def updateResultsFile(win):
    if not os.path.exists('results.json'):
        createResultsFile()

    with open('results.json', 'r') as f:
        results = json.load(f)

    with open('results.json', 'w') as f: 
        results['games'] += 1
        if win:
            results['wins'] += 1
        f.write(json.dumps(results))
    win_rate = results['wins']/results['games']
    print("Your win rate is {:2.0%}".format(win_rate))

Note that we create a fresh results file whenever an existing one isn’t found.

Once we are certain that a results file exists, we can read it back in with json.load() which automatically converts the JSON file back into the dictionary structure that we use in this program.

This function will only be called at the end of a game, so we can increment the number of the games. Then, we can check whether the last game was a win and if it was, increment the number of wins. After those two things are updated, we can write the file with the new data using the same method we used to create it. At this point, we can also show the user their updated win rate.

And that’s it! We saw how to keep track of results from this game in JSON file.

This may be the last post in the series, but if it isn’t, next time we’ll look at some more enhancements for this little game.

Making a Jamaican Creole Dictionary: Word Frequency

2021-11-29T00:00:00+00:00

Previous Post In the last post, we created a list of words in a text file that contains each word used exactly once. Since we might be interested in which words occur frequently or infrequently in our source text, we’ll look at how to associate a count with each word in our list. We’ll also look at how to create a DataFrame, which we’ll use to store other useful information about the words in the list.

To begin, let’s remind ourselves of how we created the word list in the first place:

with open('JCR_words.txt', 'w') as f:
    for book, num_chapters in book_num_chapters.items():
        for chapter in range(1, num_chapters + 1):
            url = f"https://www.bible.com/bible/476/{book}.{str(chapter)}.JNT"
            req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
            page = urlopen(req).read()
            soup = BeautifulSoup(page, 'html.parser')
            verse_elems = soup.find_all('span', class_='content')

            for verse_elem in verse_elems:
                verse_text = verse_elem.text.split()
                for word in verse_text:
                    words.append(''.join(filter(str.isalnum, word.lower())))

    words = sorted(list(set(words)))

    for word in words:
        f.write(word + "\n")

Counting Frequencies

To build on what we have, all we need to do is record the number of times each word is seen. One way to achieve this is by using a dictionary that maps a word to the number of times that it appears. If the word is already in the dictionary then we can increment its count, if not, we can add it to the dictionary with a count of 1.¹ Recall that in the last post, we normalized the words by making them all lowercase, which will avoid the problem of having capitalized words included twice.

frequencies = {}

words = []
frequencies = {}

with open('JCR_words.txt', 'w') as f:
    for book, num_chapters in book_num_chapters.items():
        for chapter in range(1, num_chapters + 1):
            print(chapter)
            url = f"https://www.bible.com/bible/476/{book}.{str(chapter)}.JNT"
            req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
            page = urlopen(req).read()
            soup = BeautifulSoup(page, 'html.parser')
            verse_elems = soup.find_all('span', class_='content')

            for verse_elem in verse_elems:
                verse_text = verse_elem.text.split()
                for word in verse_text:
                    clean_word = ''.join(filter(str.isalnum, word.lower()))
                    words.append(clean_word)
                    if clean_word in frequencies.keys():
                        frequencies[clean_word] += 1
                    else:
                        frequencies[clean_word] = 1

    words = sorted(list(set(words)))

Creating a DataFrame

Now that we have some interesting data about each word, we can start to fill in a DataFrame. This data structure allows us to store the words and some of their atrtributes in a table-like structure, which will prove useful in the future when we want to add additional data about each word. This data structure will also make it really easy to save our findings to a file that can be reused in the future so that we don’t need to make an API call every time (our text file served us well but this is more legit 😎).

DataFrame is a part of the panda module so we need to be sure to import that:

import pandas as pd
...

Then we can put the data from our dictionary into a new DataFrame:

words = []
frequencies = {}

with open('JCR_words.txt', 'w') as f:
    for book, num_chapters in book_num_chapters.items():
        for chapter in range(1, num_chapters + 1):
            print(chapter)
            url = f"https://www.bible.com/bible/476/{book}.{str(chapter)}.JNT"
            req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
            page = urlopen(req).read()
            soup = BeautifulSoup(page, 'html.parser')
            verse_elems = soup.find_all('span', class_='content')

            for verse_elem in verse_elems:
                verse_text = verse_elem.text.split()
                for word in verse_text:
                    clean_word = ''.join(filter(str.isalnum, word.lower()))
                    words.append(clean_word)
                    if clean_word in frequencies.keys():
                        frequencies[clean_word] += 1
                    else:
                        frequencies[clean_word] = 1

    words = sorted(list(set(words)))

    for word in words:
        f.write(word + "\n")

df = pd.DataFrame(list(frequencies.items()), columns = ('word', 'frequency'))
print(df)

       word     frequency
      dis          108
        a         1427
       di         1338
      lis            1
    krais           17
...      ...          ...
   iel            1
   ben            1
   kuk            1
 ilevn            1
evriwe            1

[1199 rows x 2 columns]

The DataFrame has a row for each word and its frequency.

Writing to a File

Since we have our data in something that looks roughly like a table, we might want to export it to an Excel file to do some more interesting things with, and which we can also read back into other programs.

Pandas provides a really simple way to do this:

df.to_excel('frequencies.xlsx')

And that’s it for this post. Next time, we’ll look at how we can generate an index that will tell us which book and chapter each word can be found in.

Alternatively, we could have done this all when duplicates had not yet been removed from words. ↩

Adding Multiple Rounds to The Doomsday Game

2021-11-22T00:00:00+00:00

Last time, we created a game for getting better at using the Doomsday Algorithm. After trying the game out a few times, I realized that there were a few simple but useful improvements I could make:

Allow the user to make several guesses before ending the game and printing the answer
Use only dates in the current calendar to keep the doomsday fixed (at least in easy mode).
Use a file to record the number of games and wins to allow a success rate to be printed
Add game modes that can be chosen with an argument

We’ll focus on the first improvement in this post and the other three in future posts.

All of the code for this project is available on GitHub.

Making multiple guesses

For starters, we’ll give the user three guesses to get the correct answer. Later on, we’ll change this to a default that can be overridden with a command line argument.

To implement this concept, we’ll place the logic that handles the guessing and checking inside a while loop that will repeat as long as the maximum number of guesses hasn’t been reached.

def main():
    random_date = randomDate()
    weekday = getDayOfWeek(random_date)
    print(random_date)
    
    max_guesses = 3
    guesses = 0
    
    # Allows multiple guesses
    while guesses < max_guesses:
        # Print message and read input
        guess = input("Make a guess: ")

        # Check whether the answer is correct
        if(guess.lower() == weekday.lower()):
            print("Correct!")
            return
        else:
            print("Incorrect!")
            guesses += 1

    print("Game over!")

In this code, the user can guess up to three times before the loop terminates and they see the "Game over!" message. If the user gets the answer right before that, the program exits.

Adding arguments

To add arguments, we can use the argparser module. I’ve added the argument --max_guesses using the ArgumentParser.add_argument() function. The default value is three, which is the number of guesses the user will be able to make when --max_guesses is not specified.

def main():
    parser = argparse.ArgumentParser(description='Guess the weekday for a random date')
    parser.add_argument('--max_guesses', default=3, dest='max_guesses', type=int)
    args = parser.parse_args()
    
    random_date = randomDate()
    weekday = getDayOfWeek(random_date)
    print(random_date)
    
    max_guesses = args.max_guesses
    ...

Now we can make the game sudden death by starting it with python3 main.py --max_guesses=1 or make it more lenient using --max_guesses=100 (even though I’d probably give up if it took that long 😅).

Next time, we’ll look at how we can add a feature that keeps track of game statistics.

A Doomsday Algorithm Game

2021-11-18T00:00:00+00:00

Something that I’ve been interested in for a long time but never got around to actually doing is learning the Doomsday algorithm, a method for mentally calculating the day of the week for any given date.

My curiosity was rekindled a few weeks ago when the Numberphile YouTube channel posted a video explaining how the method works. While I was looking at the Wikipedia page, I found this quote:

To improve his speed, he practiced his calendrical calculations on his computer, which was programmed to quiz him with random dates every time he logged on.

Apparently this was mentioned in a Scientific American article that, unfortunately, seems to have vanished from the internet.

I thought that an interesting project might be to try replicate that program that gives the user a date and asks them to respond with the day. I’ve decided to implement this program in Python.

All of the code for this project is available on GitHub.

Reading input

For starters, I need to set up a way to send a message to the reader and to get a response back. This code prints the message “make a guess: “ then waits for the user to type something and hit enter. After the user responds, the program prints back their response.

def main():
    guess = input("make a guess: ")
    print(guess)
    

if __name__ == '__main__':
  main()

Picking a Date

Next up is generating a random date to test the user with. Since some dates are harder than others, it makes sense to prompt the user for a difficulty at the beginning, which I’ll do later on. For now, let’s create a random date in the past year. I got some help with this from an article on Kite.

This code sets a range starting with a date a year ago and ending today. It then determines the number of days in that range and picks a day by choosing a number between zero and that number. The actual date is determined by adding that number of days to the date a year ago.

# returns a random date
def randomDate():
    end_date = datetime.date.today()
    start_date = end_date.replace(year=end_date.year-1)

    time_between_dates = end_date - start_date
    days_between_dates = time_between_dates.days
    random_number_of_days = random.randrange(days_between_dates)
    random_date = start_date + datetime.timedelta(days=random_number_of_days)
    print(random_date)

Finding the Day

Now that we have a date, we need to determine what day of the week it was before we prompt the user. Thankfully, this is built into the datetime.date object under the .weekday() function.

def main():
    random_date = randomDate()
    weekday = random_date.weekday()
    print(random_date, weekday)

The weekday function of datetime.date returns a number, so to make things easier, we can convert this number to a string using date.strft() (which means we really could have skipped using date.weekday(). I’m also going to move this logic into a function that takes the date object and returns the string representation of the weekday.

def getDayOfWeek(date):
    return date.strftime('%A')

def main():
    random_date = randomDate()
    weekday = getDayOfWeek(random_date)
    print(random_date, weekday)
...

Checking the Answer

Now that we know what the correct answer is, all that’s left to do is to check whether the user’s answer is correct. To keep things simple, I’ll make the correct answer and the user’s answer lowercase before I convert them.

def main():
    ...
    if(guess.lower() == weekday.lower()):
        print("Correct!")
    else:
        print("Incorrect!")
    if(guess.lower() == weekday.lower()):

And that’s it for now! In my next post I’ll introduce and implement some ideas for improvements.

Making a Jamaican Creole Dictionary

2021-11-11T00:00:00+00:00

Recently there has been an ongoing effort by several groups to legitimize Jamaican Creole as an official language. These efforts have included the translation of the New Testament of the Bible into that language. Since I’m involved with a group that has adopted and is actively promoting the use of Jamaican Creole, I decided that it might be useful to create a “dictionary” by compiling a list of all the words that appear in this translation. This post describes the process I went through to achieve this as well as the code that I wrote, which is in this repository on GitHub. The repository also includes a colab notebook where you can follow along.

Prerequisites

Di Jamiekan Nyuu Testiment

I wasn’t able to find a plain text copy of this translation so part of this project involved reading the text off of the bible.com website, where the translation is freely available. I’ve made the text I parsed available in the repository for this project.

Python

I chose to use Python for this project because I felt that it lends itself to working with text.

BeautifulSoup

I used BeautifulSoup to work with the HTML that I read from bible.com. It can be installed with pip using python3 -m pip install bs4.

Getting Started

The bulk of this project involves simply getting the raw text from bible.com. To achieve this I used urllib.request which provides an interface for creating an HTTP request.

To actually build the request, I needed to be able to request each chapter of each book of the text individually because the site displays each chapter as a page. Since I couldn’t think of a better way to determine which chapters should be requested, I stored the number of chapters in each book in a dictionary that maps the abbreviation of the book name that bible.com uses in its url to the number of chapters it has. I could then interpolate the url with the abrreviated book name and each chapter to make the request. The snippet below shows how to make the request for the first chapter of Di Jamiekan Nyuu Testiment:

from urllib.request import Request, urlopen

book = 'MAT'
chapter = 1

url = f"https://www.bible.com/bible/476/{book}.{str(chapter)}.JNT"
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
page = urlopen(req).read()

For the rest of this post, I’ll continue with this book and chapter but the process can be generalized by iterating over the key, value pairs in the dictionary described above and then making a request for each chapter from chapter 1 to the number of chapters in that book.

Now that we have made the request and stored the response in page, we can begin to use BeautifulSoup to work with the HTML. At the moment, page is just a soup (😉) of characters, so BeautifulSoup will be really handy for making sense of it.

The first step in parsing will be to create a BeautifulSoup object from the HTML string, which we’ll call soup. This will then provide us with some useful functions for finding elements in the HTML.

from bs4 import BeautifulSoup

soup = BeautifulSoup(page, 'html.parser')

After some quick inspect-elementing on bible.com, I realized that every verse was wrapped in a span element with the class _content. There were also some blank elements that appeared with this structure, but since they were blank anyway, I just ignored them.

To select these elements, we use the soup’s find_all function which takes a tag name and some properties and returns a list of elements that match those criteria.

verse_elems = soup.find_all('span', class_='content')

We can now iterate over each of these elements and get their inner text using the .text property of the BeautifulSoup Tag object, which is how the HTML elements are represented.

verses = [verse.text for verse in verse_elems]

Now that we have the text of each verse, we can split it into words and add them to our array¹. I used the filter function to only include alphanumeric characters in the words I added and the .lower() function to make sure that capitalized words weren’t counted seperately.

words = []

for verse in verses:
    for word in verse.split():
        words.append(''.join(filter(str.isalnum, word.lower())))

Finally, to get the full list, I can cast the words list to a Set and then back to a list to remove duplicates and then use sorted to put the words in alphabetical order.

words = sorted(list(set(words)))

We can check how many words were found using len(words) (I got 169).

With the list finalized, we can now write it to a file, which will have each word on a line:

with open('JCR_words.txt', 'w') as f:
    for word in words:
        f.write(word + "\n")

And that’s it! We now have a list of all words in Matyu 1 and we can generalize it to all books and chapters.

In my next post, I’ll go through how we can use the generated text file to create a frequency count and index.

I definitely could have used a set here but I used an array because I wanted to preserve word frequencies for the time being. ↩

Welcome to My Blog!

2021-11-10T00:00:00+00:00

Welcome to my first blog post! My plan for this website is to post about things that I’m working on (probably mostly CS-related or adjacent) every week.

I’ve wanted to do something like this for a while but the way I had my website configured for the past year or two just wasn’t conducive to adding blog posts. I’d built the website using React, so there was a significant amount of overhead to add new posts to my page. Arguably, I could have probably found a way to make the process more fluid and automated but I just never got around to it.

In the past year or two, I started learning much more about Emacs org-mode. I’d used the org files in a very rudimentary way previously but I was able to dive into the manual over the summer of 2020 and that really opened up my eyes to how feature rich that package is (my favorite is still exporting to PDFs that are rendered with LaTeX). One thing I stumbled across that stood out to me was that org can export your org files to HTML which can then be served with Jekyll. Since at the time I was (and still am) using GitHub pages to host my website (at least this blog portion for now) it seemed like a really easy way to build a blog without the headache of writing all the HTML myself.

I ended up running into quite a few roadblocks getting that setup to work smoothly, so I decided that it would be best to get the site running with vanilla Jekyll and then revisit the idea of publishing directly from Emacs later on.

So for now, I’ll be writing these posts in markdown while I sort out that org mode stuff. Honestly, Jekyll seems to be more than enough for my use case, but I also like fiddling with Emacs, plus I want to move my task tracking back to org mode/org agenda/org journal after trying out Notion for the past year or so, but that’s for another post 😉

In general, I plan to start by codifying the existing ideas and projects I’ve been working on and then hopefully move on to some new stuff 😃

Useful Links:

The tutorial I followed for GitHub pages/Jekyll setup