Over the course of a night and morning I built a small flask app (Flask is a micro web framework for Python. Django is similar but more heavy-duty). It is essentially a dictionary. You can search for an entry by its keyword, or view the helptexts for all the entries, or (if you are logged in) add and remove entries. It is a straightforward derivative of excellent the Flaskr tutorial.
For me, I am really proud of this app even though it is dirt simple and not much to look at because this is the first time I really interacted with a database in a programming language.
The intent was to make a database of documentation for the AVR Atmega328P microcontroller. However, since then I have decided to move to using a mediawiki (sample page).
The app may be seen live at salty-retreat-5363.herokuapp.com. The username is “admin” and the password is “default”. The database gets wiped at least once every 24 hours.
If you’d like to run the app locally and play around with source code, please see https://github.com/nouyang/WTFisThisRegister for detailed instructions. That page also details how to deploy this app (or similar ones, such as Flaskr) to Heroku. I used http://dillinger.io/ to play around with markdown online and learn it and actually document this project with a proper README.
Things Learned
Here are some things I learned in the course of making this:
- Gitignore files need to be created before you run “git add .” — in general, the line telling it to ignore a file needs to exist before you added it to the version control, git won’t automagically remove it for you.
- Heroku DOES play with sqlite3 but not happily, for instance the database will be wiped at least once every 24 hours. See https://devcenter.heroku.com/articles/sqlite3.
- Virtualenv does NOT like spaces in your directory path! This was tricky to debug as even in -v verbose mode the full path for my virtualenv packages was shortened and so I couldn’t even notice that my path had spaces in it.
- Databases use cursors.
The Process
Here are some images from the process. Click on them to view them in full resolution.
1. First I started out just using a dictionary (no database involved) and a single hello.py file.
2. I then followed the Flaskr tutorial. After doing so I wanted to modify it so that the text was monospaced and preserved whitespaces, because it doesn’t by default.
I used the “pre” tag at first.
but soon switched to using CSS.
This actually caused me a lot of headache later troubleshooting why there was random whitespace in my design, because I didn’t restrict where the CSS was applied enough. So much whitespace:
In the end I had this file: https://github.com/nouyang/WTFisThisRegister/blob/master/static/style.css
.helptext { preserve whitespace with wrapping CSS }and the corresponding code in the html file was
<div class="helptext"%gt;
{{ entry.helptext|safe }}</div>
3. I wanted to be able to delete entries, so I added a delete button.
In show_entries.htmloops can’t show it because blogger can’t escape this HTML properly :/ In the main python file:
@app.route('/delete', methods='POST'])
def delete_entry():
if not session.get('logged_in'):
abort(401)
g.db.execute('delete from entries where keyword = ?', [request.form['entry_to_delete']])
g.db.commit()
flash('Entry was successfully removed')
return redirect(url_for('show_entries')) The “?” is a safe why for sqlite3 to accept text (in this case, whatever was entered in the form with the name entry_to_delete).
4. I learned to play around with the database to test my SQL commands in the terminal:
$ sqlite3 /tmp/flaskr.db
sqlite> select helptext from entries where helptext='PCMSK0'5. Finally I added a search functionality to the site:
@app.route('/search', methods=['POST'])
def search_entries():
keyword = request.form['searchterm']
cur = g.db.execute('select helptext from entries where keyword = ?',
[keyword])
result = [dict(keyword=keyword, helptext=row[0]) for row in cur.fetchall()]
return render_template('search.html', result=result)
And set the homepage to be the search page:
@app.route('/')
def show_search():
return render_template('search.html') 6. I learned about initializing databases, namely by dumbly thinking I needed to include the flaskr.db file and no schema.sql file in the repository to allow other people to get it working on their systems. Actually, I was just missing a line, “init_db()”, before “app_run()”. In the end I am still keeping it without that line so that I have persistence in the database (I can stop the server and restart it and the database entries will still be there).
All in all I learned a lot! Do browse through the source code and play it with yourself if you’d like.
=========================================
Liveblog Notes (kept here for reference)
http://flask.pocoo.org/docs/quickstart
http://maximebf.com/blog/2012/10/building-websites-in-python-with-flask/
https://github.com/trtg/flask_assets_tutorial
(did not use: http://www.realpython.com/blog/python/python-web-applications-with-flask-part-ii-app-creation/)
http://flask.pocoo.org/docs/tutorial/introduction/ Ah! Here’s the hands-on quickstart.
Hmm, well now the newlines show up but long lines aren’t wrapped.
To fix:
http://stackoverflow.com/questions/642140/retain-newlines-in-html-but-wrap-text-possible
.page { width: 70%;
min-width:35em;
border-radius:5px;
}
Modifying the database name: How are you supposed to properly do this? I just deleted /tmp/flaskr.db and then ran in the shell:
sqlite3 /tmp/flaskr.db < schema.sql
$ python>>> from flaskr import init_db>>> init_db()
select helptext from entries where keyword=”PCMSK0″;
http://www.zetcode.com/db/sqlitepythontutorial/
http://docs.python.org/2/library/sqlite3.html
Let’s install the AVR eclipse plugin
http://avr-eclipse.sourceforge.net/wiki/index.php/The_AVR_GCC_Toolchain#Debian_and_Ubuntu
http://www.instructables.com/id/How-to-get-started-with-Eclipse-and-AVR/step13/Quick-Tour-Of-Cool-Features/ (windows)sudo apt-get install eclipse (270+ MB! sigh)
sudo apt-get install gcc-avr binutils-avr gdb-avr avr-libc avrdudehttp://avr-eclipse.sourceforge.net/wiki/index.php/Plugin_Download
Heroku deploy
http://flask.pocoo.org/docs/quickstart/#quickstart-deployment
https://devcenter.heroku.com/articles/getting-started-with-python
- Install heroku toolchain
- Login to heroku
- Install and activate virtual environment
- Install dependencies (why do I need gunicorn? not sure)
- Make a Procfile
- Run foreman and check in browser that app is up on localhost
- Download the python .gitignore and add the “venv” line to the top
- my own step: add to github, git remote add origin https://github.com/nouyang/WTFisThisRegister.git
https://help.github.com/articles/ignoring-files
OH. The .gitignore file does NOT go in the root directory directly. No wonder everything failed and I just had to delete my github repo and also the .git folder in my root directory.
Instead,
For example, you might create the file at ~/.gitignore_global and add some rules to it. To add this file to your cross-repository configuration, run
git config --global core.excludesfile ~/.gitignore_global.Local per-repository rules can be added to the.git/info/excludefile in your repository. This method can be used for locally-generated files that you don’t expect other users to generate.
Wait, but what if there’s one file, e.g. a passwords for the database file, that I don’t want to commit? Maybe it’s just messed up because I did the git init and then added the gitignore file or something funky, I don’t remember. http://stackoverflow.com/questions/1139762/gitignore-file-not-ignoring
Okay, so now I want to pull the config parameters (passwords, admin usernames, secretkeys) out into a separate file that is gitignored.
Done. Created a file called databaseconfig.py, added the line “from databaseconfig import secretkey, username, password” to the main python file.
Oh I should generate a real secret key
http://stackoverflow.com/questions/18247971/print-python-os-urandom-output-on-terminal
python
>>> import os, binascii
>>> binascii.hexlify(os.urandom(24))
Whoa they actually mean create the file ~/.gitignore_global, not create ~/.gitignore_global/.gitignore
Whoops.
http://stackoverflow.com/questions/4824188/git-ignore-vim-temporary-files
Also, vim and gedit create swap files not covered by github’s gitignores, so add
*~and
*.swp
*.swoto ~/.gitignore_global
Yes! Succesfully used .gitignore for once.
(venv)nrw@nrw-PC:~/projects/WTFisThisRegister$ git commit -m “init”
[master (root-commit) 35b1400] init
10 files changed, 228 insertions(+)
create mode 100644 .gitignore
create mode 100644 Procfile
create mode 100644 README.md
create mode 100644 WTFisThisRegister.py
create mode 100644 requirements.txt
create mode 100644 static/style.css
create mode 100644 templates/layout.html
create mode 100644 templates/login.html
create mode 100644 templates/search.html
create mode 100644 templates/show_entries.html
git remote add origin https://github.com/nouyang/WTFisThisRegister.git
git push -u origin masterheroku create
git push heroku master
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
Whoops I need to add this computer to heroku.
https://devcenter.heroku.com/articles/keys
$ ssh-keygen -t rsa
/home/nrw/.ssh/id_rsa already exists.
—–> Launching… done, v3
http://salty-retreat-5363.herokuapp.com deployed to Heroku
http://stackoverflow.com/questions/18037027/flask-app-failing-on-heroku-but-working-with-foreman
S3_KEY=mykey
NOT
S3_KEY = mykey
Note: Only today I realized that in pasting URLs from the location bar in chromium-browser (ubuntu 12.10) into blogger, the URLs are automatically turned into hyperlinks, but they remain plain old text and I have to manually turn them into hyperlinks when I copy them from firefox, my main browser 🙁 Such a weird specific bug.
Okay, let’s look at the heroku logs.
https://devcenter.heroku.com/articles/logging
ImportError: No module named databaseconfig
The solution is to use os.environ.get() instead.
Okay, so I’ll just write a short if statement checking if os.environ.get() is None.
SECRET_KEY = os.environ.get(‘SECRETKEY’) #for heroku deploys.
if SECRET_KEY == None:
SECRET_KEY = secretkey
Now to get databaseconfig to show up in heroku. Do I want to?
No. A cleaner solution instead of all this is to use try, importerrors. Since the present of databaseconfig.py can be used to indicate that we are running it locally.
ahhhh now the /entries works locally but there is an internal server error remotely. WHY.
http://stackoverflow.com/questions/8950674/debugging-a-flask-app-running-in-gunicorn
Instead of foreman start, just use foreman run python app.py if you want to debug your application in development.
Maybe it’s an issue with gunicorn or procfile or whatever that stuff is?
web: gunicorn WTFisThisRegister:app
http://ryaneshea.com/lightweight-python-apps-with-flask-twitter-bootstrap-and-heroku
changed to
web: python WTFisThisRegister.py
Okay, still works locally but not remotely. Maybe it is a port issue. Sucks that these tiny changes debugging production (heroku environment) take forever to try (30 seconds for each time I git push heroku master)
2014-01-12T05:58:48.956357+00:00 heroku[web.1]: State changed from crashed to starting
AGH. Still internal server error. WTF.
http://stackoverflow.com/questions/7653010/import-sqlite3-with-python2-7-on-heroku
So sqlite3 could NOT be used on the previous version of heroku. However, I think it should be able to now?
https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
http://stackoverflow.com/questions/7784471/using-sqlite3-on-heroku-cedar-stack
CANNOT. Ugh. So this is all pointless. Time to migrate to postgreSQL or something that supports sqlite3.
Sigh. I guess I’ll just submit a documentation issue about this whole fiasco.
https://github.com/mitsuhiko/flask/issues/949
Okay, looking into how other people deployed their flaskr projects.
https://github.com/mjhea0/flaskr-tdd
This person has a LOT of documentation and talks about jquery / AJAX. Todo for the future!
https://github.com/mjhea0/flaskr-tdd/blob/master/app.py
WAIT. WTF. It looks like you can use sqlite3, I’m just dumb and you have to include the databse in your root directly, not off in /tmp/ somewhere.
So, looks like DATABASE = ‘flaskr.db’
sqlite3 ./flaskr.db < schema.sql
Note that the way I have it set up — foreman locally will run with Debug = False, heroku will run with Debug = false, and python locally will run with Debug = True. A rather peculiar state of affairs, but fine for me.
Let’s debug the download-from-github install for the Readme.
$ virtualenv –distribute –no-site-packages venv
The –no-site-packages flag is deprecated; it is now the default behavior.
New python executable in env/bin/python
Installing distribute………………………………………………………………………………………………………………………………………………………………………done.
Installing pip…
Error [Errno 2] No such file or directory while executing command /home/nrw/projects/t…env/bin/easy_install /usr/share/python-vi…p-1.1.debian1.tar.gz
…Installing pip…done.
http://stackoverflow.com/questions/10068388/virtualenv-returning-a-no-such-file-or-directory-error
sudo apt-get remove python-virtualenv
sudo pip install virtualenv
bash: /usr/bin/virtualenv: No such file or directory