Editing Twitter Analysis DB

Jump to navigation Jump to search

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Latest revision Your text
Line 1: Line 1:
 
= Goal =
 
= Goal =
This document could use improvement, but the software largely works ( see Status below ) and I am probably done unless there is expressed interest in knowing more, or I add major new features. Look at history tab to see what is going on in the document. If you would rather look at at the application than read about it see [[Twitter Analysis DB GUI]] which you should check out at some point in any case.
+
This document just started ... will probably take a week or so  look at history tab to see what is going on  
  
Twitter Analysis DB is a Python open source, program and an accompanying database, running in a Graphical User Interface tool ( and/or database creation tool ) for the analysis of a body of tweets.  Currently the program is in early alpha and its design goals are evolving at least as fast as the code is being written.
+
Twitter Analysis DB is a Python open source, program and an accompanying database ( and/or database creation tool ) for analysis of a body of tweets.  Currently the program is in early alpha and its design goals are evolving at least as fast as the code is being written.
  
The point:
+
This program is designed to be:
* Quickly select and view tweets ( typically for a single person based on a number of criteria ).
 
* Display concordance and simple linguistic analysis for a body of tweets.
 
* Provide a database of tweet and concordance data for analysis using tools external to the app.
 
 
 
 
 
This program has some supporting functionality:
 
 
* Load a body of tweets into a database ( currently sqlite )
 
* Load a body of tweets into a database ( currently sqlite )
 
* Break the tweets down into a concordance.  
 
* Break the tweets down into a concordance.  
* Have an accompanying db of English words ( right now sourced from Kaggle )
+
* Have an accompanying db of English words ( I need to find out how to provide within copywrite guidelines )
 
* Provide a variety of queries against the database for possible enlightenment.
 
* Provide a variety of queries against the database for possible enlightenment.
  
 +
I  will try to documented well enough so people can relatively easily extend and adapt the program.  Or as alternative they can use other tools with the database like SQLiteStudio. It should be fairly easy to download and use even for those without a desire to dive into the code but, I assume some knowledge of Python, and a Python Environment to run it in.  In Python 3.6 or so.  See more features below.?
  
I  will try to documented well enough so people can relatively easily extend and adapt the program.  Or as alternative they can use other tools with the database like SQLiteStudio. It should be fairly easy to download and use even for those without a desire to dive into the code but, I assume some knowledge of Python, and a Python Environment to run it in.  In Python 3.6 or so. 
+
See the graphical user interface here ( with screen shot ): [[Smart Terminal GUI]]
 
 
See the graphical user interface here ( with screen shot ): '''[[Twitter Analysis DB GUI]]'''.
 
  
 
This application is also part of a family of applications see the category below: Python Projects.
 
This application is also part of a family of applications see the category below: Python Projects.
  
This is an article started by Russ Hensel, see '''"http://www.opencircuits.com/index.php?title=Russ_hensel#About My Articles"'''    About My Articles for a bit of info.
+
This is an article started by Russ Hensel, see "http://www.opencircuits.com/index.php?title=Russ_hensel#About My Articles" About My Articles for a bit of info.
 
 
Code will be at GitHub, see '''[[https://github.com/russ-hensel/twitter_analysis_db Code at GitHub]]'''  See the GUI here at '''[[Twitter Analysis DB GUI]]'''
 
 
 
== Status ==
 
 
 
* It works....typically produces output consistent with user input.
 
* Overall structure seems sound and extensible.
 
* Should be relatively easy to add additional queries, joins, columns, select criteria, without massive coding effort.
 
* But.... it is full of opportunities for enhancement.  Right now my interests have shifted so I may not do much further work.  Possibilits for improvement:
 
** Clean up tweet in the database build stage.  Pretty good not but still some odd "words" get through.
 
** User interface is evolving but still not as user friendly as I would like.
 
** Selects == (also know as Reports or Queries... ) are more demos of what is possible than what is truly useful and informative, several are experiments in the technology of the application.
 
** Sqllite still doing ok at 4 years of trump tweets and 300k of words. 
 
** No database optimizations yet.... I run on ram drive for speed. DB is about 40 MBytes with 4 years of trump tweets
 
** Report formatting is basic, but workable.  Nicest overall format for human readability is probably "html", best to pass to other applications is probably "csv", most responsive in time is "msg"  -- sent to message area, often sub second response.
 
** Not sure what area of work is most useful, have been driven lately by programming challenges need to focus for a bit on improving usefulness.
 
** Still printing some unnecessary junk used in debugging, remove most... if output is needed sent to py_log, but whole logging parts of the application could use a careful review ( not happening soon ).
 
  
'''What technical knowledge should users have ( and How ):'''
+
Code will be at GitHub, see [[https://github.com/russ-hensel/python_smart_terminal Code at GitHub]]  See the GUI here at [[ Twitter Analysis DB GUI]]
 +
'''Who should use this program and How:'''
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 53: Line 30:
 
|Person with little programming experience, no interest in Python.  Looking for download and install.
 
|Person with little programming experience, no interest in Python.  Looking for download and install.
 
|Probably should use another program.
 
|Probably should use another program.
|Not well suited to use this, but I may try to build an exe at some point.
+
|Not well suited to use this, I do not plan to fix this.
 
<!-------------------------------->
 
<!-------------------------------->
 
<!-------------------------------->
 
<!-------------------------------->
 
|-valign="top"
 
|-valign="top"
|Programming in some text based language, willing to install or has Python installed.
+
|Programming microcontrollers in text based language, willing to install or has Python installed.
 
|Run the basic program.  Customize in fairly minor ways.
 
|Run the basic program.  Customize in fairly minor ways.
|Edit the parameter file to configure the terminal.   
+
|Edit the parameter file to configure the terminal.  Could use processing extensions written by others.
 
<!-------------------------------->
 
<!-------------------------------->
 
|-valign="top"
 
|-valign="top"
 
|Modest Python experience
 
|Modest Python experience
|Modify all over the place, save data to database ......
+
|Modify all over the place, monitor equipment, save data to database ......
|Program should be well documented in source, with some supplement in this wiki, or ask the author.
+
|Program should be well documented with an API, sample extensions.  Code changes localized ( now ) to one class plus the parameter file.
 
<!--------------------------------
 
<!--------------------------------
 
|-valign="top"
 
|-valign="top"
Line 91: Line 68:
 
* Runs across OSs Linux ( inc '''Raspberry Pi''' ), Mac or Windows  
 
* Runs across OSs Linux ( inc '''Raspberry Pi''' ), Mac or Windows  
 
* Python
 
* Python
 +
 +
 
* Multiple tables, preloaded with sample data.
 
* Multiple tables, preloaded with sample data.
 
   
 
   
* Database Interface ( sqlite, could be modified for other relational databases )  
+
* Database Interface ( sqlite, could be modified for other relational databases )
 +
 
* Parameter file for wide range of modifications of program behavior.
 
* Parameter file for wide range of modifications of program behavior.
 
* Uses standard Python logging class.
 
* Uses standard Python logging class.
 +
 
* For more on the features see [[???]]
 
* For more on the features see [[???]]
  
Line 117: Line 98:
 
== Download ==
 
== Download ==
  
Code coming at GitHub, see [[https://github.com/russ-hensel/twitter_analysis_db GitHub Repository]] ( it is Python and you can run directly from the source ) Email me if you have issues ( use this link [[User:Russ_hensel]] ).
+
Code comming at GitHub, see [[https://github.com/russ-hensel/python_smart_terminal Code at GitHub]] ( it is Python and you can run directly from the source ) Email me if you have issues ( use this link [[User:Russ_hensel]] ).
 
You will get a zip file, unzip it and you should get:
 
You will get a zip file, unzip it and you should get:
  
Line 123: Line 104:
 
     .... whatever --|
 
     .... whatever --|
 
                     |
 
                     |
                     |-- twitter_analysis_db --- all code required to run the application ( not sure if smart_terminal or python_smart_terminal or nothing is top level name, just put it in some well named place )
+
                     |-- tbd----------- --| -> all code required to run the application ( not sure if smart_terminal or python_smart_terminal or nothing is top level name, just put it in some well named place )
                                              |    some logs from my running of the code may or may not be present, these will be deleted as they creep in, when you run the program you will
+
                                          |    some logs from my running of the code may or may not be present, these will be deleted as they creep in, when you run the program you will
                                              |    get your own log files ... all typically named xxx.py_log  
+
                                          |    get your own log files ... all typically named xxx.py_log  
                                              |
+
                                          |
                                              | --> input      ---  input files used to build the database.
+
                                          | -- images -> image files, mostly screen shots, icons... or what ever, not important for the code.
                                              | --> output    ---  files produced by the database selects.
+
                                          | -- wiki_etc -> various files documenting program, including at least some of the material from this wiki  
                                              | --> images    ---  image files, mostly screen shots, icons... or what ever, not important for the code.
 
                                              | --> wiki_etc   ---  various files documenting program, including at least some of the material from this wiki  
 
                                              |                    also some sample output files
 
                                              | --> help      ---  help files, documentation for various selects
 
                                              | --> resources  ---  source code for the HTML module used in the application see [[Python Installation]]                                         
 
  
Put them in your system making "....whatever" anything convenient for your Python installation ( that is move the files to where you keep your Python source, not your installed module location ).   
+
Put them in your system making "....whatever" anything convenient for your Python ( that is move the files to where you keep your Python source ).   
  
Note that there may be a certain amount of left over, dead code, in the directories I am cleaning out bit by bit, someday it may be nice and neat.  For now if you want to tinker look at the design info below first.
+
Note that there may be a certain amount of left over, dead code, in the directory I am cleaning out bit by bit, someday it may be nice and neat.  For now if you want to tinker look at the design info below first.
  
I have not yet made a requirements.txt or any installation routines.  Run as you would any source code until all imports work.  A couple of modules ( '''spacy''' and ''HTML'' ) proved a bit difficult.  I have directions for these in: '''[[Python Installation]]'''.
+
== Run ==
  
== Run ==
+
Run it until it stops complaining about dependencies ( in the console ), after that ( and perhaps even before ) the GUI should come up.  You are installed.
  
Run it until it stops complaining about dependencies ( in the console ), after that ( and perhaps even before ) the GUI should come upYou are installed. ( Also see note above about '''[[Python Installation]]'''. )
+
I have run the program on both Windows 10 and Rasperian on a RPi.  It should work in most OS'sLet me know about issues.
  
I have run the program on both Windows 10.  It should work in most OS's but this is untestedLet me know about issues.
+
Some parts of the program think that mySql is available.  It should run fine without it ( there will be a message or two in the console ), untill you try to use the database then it is not so graciousStraight ahead use as a terminal does not use the database.
  
 
= Configure to Run =
 
= Configure to Run =
Line 157: Line 133:
 
Parameters starts out with some "meta" parameters.  These are defined early in the creation of the objects and may effect other values.  In any case you can always define a value twice, the last one always wins.  The most important meta parameter is mode, you should not change it from self.mode = "working on this" unless you understand the implications or do not mind going on a ride.
 
Parameters starts out with some "meta" parameters.  These are defined early in the creation of the objects and may effect other values.  In any case you can always define a value twice, the last one always wins.  The most important meta parameter is mode, you should not change it from self.mode = "working on this" unless you understand the implications or do not mind going on a ride.
  
=
+
== Command Line Arguments ==
 +
 
 +
If you run with the command line parameters=paramaters_b then after the regular parameters file runs, then the system looks for parameters_b.py and uses that to override values that you might want to tweak ( or completely redo.  There are two examples in the directory follow the pattern in them and you should be fine ).
 +
 
 +
This can be especially useful if you want to run two copies connected to different ports and possibly running in different modes.  In this case it is also nice to change ( its in parameters ) the '''icon''' and '''color''' for each instance of the program.  You can write or use the little bat file to start them ( although this leaves a dos console hanging around )  Command line arguments can also be placed in shortcuts.  In either case they may take some tweaking to run in/from your file locations.
 +
 
 +
The above mostly applies to Windows, but the program run fine with Linux ( including the Raspberry Pi ) and I suppose the Mac.  Of course the .bat file and shortcuts will not work, but similar facilities exist in the other OSs.
  
 
== Running It ==
 
== Running It ==
  
When you run it it should open a windows a lot like the picture [[Twitter Analysis DB GUI]].  Errors may show up in your Python console or the log file ( look in parameters.py for the name of the log file, typically self.pylogging_fn = "tweet_app.py_log" ).  The most likely errors will point to missing Python modules like pyperclip.  You should install with pip ( or conda if using Spyder ).  Let me know how it goes.
+
When you run it it should open a windows a lot like the picture [[Smart Terminal GUI]].  Errors may show up in your Python console or the log file ( look in parameters.py for the name of the log file, typically self.pylogging_fn     = "smart_terminal.py_log" ).  The most likely errors will point to missing Python modules like pyserial.  You should install with pip ( or conda if using Spyder ).  Let me know how it goes.
  
For info on general use of parmeters.py see [[Configuration Files For Python]]
+
Normally the terminal does not open the com port until you press the <Open> button.  The parameters are displayed in the GUI if you do not like them ( for example when you press the <open> button the port open status changes to "open failed", not the desired "open" ) you can shutdown, edit parameters.py and restart.  There is a simpler way.
  
*An early setting to configure in parameters.py to set the name of a text editor on your system. This is important for working with text files including the .txt output files and the parameters.py file.  For mine this is one of:  
+
*First configure parameters.py to know the name of a text editor on your system. For mine this is one of:  
 
<pre>
 
<pre>
  
Line 178: Line 160:
 
*Now when you run it the button <Edit Parms> should let you edit the parameters.py file.  Edit it and save.
 
*Now when you run it the button <Edit Parms> should let you edit the parameters.py file.  Edit it and save.
  
*Hit the <Restart> button and in a flash ( more or less ) the program should restart with the new parameters, starting is fast because previously imported material does not need to be re-imported.
+
*Hit the <Restart> button.  In a flash the program should restart with the new parameters.
  
  
..... more here soon ?......
+
..... more here soon ......
  
 
= Notes on the Code =
 
= Notes on the Code =
Line 191: Line 173:
 
Description of the design, and a bit to help you figure out what the files do ( for .py files also see the top of file ).
 
Description of the design, and a bit to help you figure out what the files do ( for .py files also see the top of file ).
  
see: [[Twitter Analysis Technical]]
+
see: [[Python Smart Terminal Technical]]
  
 
= Customizing/Extending =
 
= Customizing/Extending =
  
Simple customization may be done simply by changing the parameter file, for other stuff you will need to wade into the code. I try to name, factor, and comment well, but it is a work in progress.
+
Simple customization may be done simply by changing the parameter file, see: [[Smart Terminal Parameter Examples]]  If you want to add code that can be done by messing with any of the source code ( not recommended ) or by programming an extension, see: [[Python Smart Terminal Technical]] and [[Writing You Own Extensions to SmartTerminal]]
  
 
= Additional Info =
 
= Additional Info =
  
*Click on the category smart Twitter Analysis DB ( and perhaps the others as well )
+
*Click on the category smart terminal below ( and perhaps the others as well )
+
*For a graphing ( plotting ) application that is a companion to this see [[Python Smart Terminal Graph]]
 
 
*[[Twitter Analysis DB GUI]]
 
*[[Twitter Analysis DB Details]]
 
*[[Python Desk Top Applications]]
 
*[https://github.com/russ-hensel/twitter_analysis_db GitHub Repository]
 
*[[Python Installation]]
 
 
 
<!-----------
 
this cannot bee seen
 
 
*[[Writing You Own Extensions to SmartTerminal]]
 
*[[Writing You Own Extensions to SmartTerminal]]
 
*[[Debugging the Smart Terminal]]  
 
*[[Debugging the Smart Terminal]]  
 +
*[[Smart Terminal as Smart Data Logger]]  obsolete, gone, may come back.
 +
*[[Smart Terminal GUI]]
 +
*[[GreenHouse Monitor Program]]
 +
*[[Smart Terminal Parameter Examples]]
 +
*[[Python Desk Top Applications]]
 +
*[ https://github.com/russ-hensel/python_smart_terminal python_smart_terminal at git hub]
 +
*[http://www.instructables.com/id/Python-Terminal-for-Cheap-Pi-Arduino-Connection/ Python-Terminal-for-Cheap-Pi-Arduino-Connection/] Info on much earlier version may or may not be useful.
  
 
+
[[Category:Python Projects]] [[category:Python]] [[Category:Arduino/RaspberryPi]] [[Category:SmartTerminal]] [[Category:Serial Communications]]
------------->
 
 
 
 
 
 
 
[[Category:Python Projects]] [[category:Python]]   [[Category:Twitter Analysis DB]]
 

Please note that all contributions to OpenCircuits may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see OpenCircuits:Copyrights for details). Do not submit copyrighted work without permission!

Cancel Editing help (opens in new window)