[Scons-users] database builder

[Philipp Kraus]

Am 13.04.2013 um 22:24 schrieb Bill Deegan:


> Phil,

> 

> There are a number of python based database migration packages.

> For example for Django there is south. (And actually it might not be tied to django, not sure).

> 

> If I understand correctly, you're looking for a facility to do both:

> a) do initial configuration of a database,it's tables,etc.

> b) upgrade/update an existing database,tables,etc


Is this the tool? https://www.djangoproject.com/ ?
I have take a look to http://www.sqlalchemy.org

But yes, I'm searching for something like this, but my full buildprocess
works with Scons, so it can be interessted to do the buildprocess also with Scons.

It is an idea at the moment, because I use Scons for creating binaries from C++ code,
a client and a server program. The client program is not a problem, but the server program
must be compiled on the server machine (a HPC node). Also on the cluster runs a database
which must be also updated / created.
In our case we have got only a SSH account without installing packages, so I know my database
login and I can run the installed C++ compilers and Python with a local Scons installation.
You can installed different buildtools, but did I need different tools? One tool that can handle
the full buildprocess should be nice.

So did you think database installation / update process can be handle with Scons with
a builder or something else? Is this a good choice? I know other tools, but in some
cases they can not be used. 
My idea is describe the database like a Python dictionary and push the dict to a builder,
after that the builder creates or updates the database (connection information can be passed
from a parameter of the SConstruct).
I don't want to do database administration from a client (with extra update code) and a native plain text SQL
script isn't always a good solution. Scons is my buildsystem and why it can not be used for database
installation?
Scons does not need any database code, but a wrapper for a Python library can be a nice feature


Phil




> 

> On Sat, Apr 13, 2013 at 7:15 AM, Philipp Kraus <philipp.kraus at flashpixx.de> wrote:

> Hi Gary,

> 

> 

> Am 13.04.2013 um 15:51 schrieb Gary Oberbrunner:

> 

>> While this is a cool idea, I don't think SCons would actually be that helpful.  I'd start by looking at how Rails migrations work.  Schema updates should be defined by the db developer; I don't think trying to diff schemas (with SCons or any tool) will actually get you where you need. 

> 

> At the moment I do this like env.Command("database", "mysqlfile.sql", "mysql.... $SOURCE")

> It works, but IMHO it's not a good solution, because the parameters can be different on different OS / versions, so I need different switches for calling the database client.

> Bruce has describe Liquidbase, so in this case for the developers of a project can be this a solution, but I would like to use Scons also on update procedure on

> the userside. But in this view I can not tell the user "you need to install liquidbase and scons and database driver.... for updating your database", also on server

> systems there are often installed not all packages eg Java, Python. Some packaging programs with GUI are don't run in a console.

> So "one tool (Scons) for everything" can be a nice idea, but for database there must be more abstraction, I think about UML specification and the model transformation.

> 

>>  There's often queries that need to be run when updating a schema for instance, to compute values for a new column or set the default value or other things.  

> 

> Yes eg you set a field to a "non null" before update it can store null values, so I must set in a preprocess the values to other values. So IMHO a short call eg alter table

> is not enough, so there must be more processing.

> The update process of a database is also a dependency graph, because the calls of changes are ordered. 

> 

> 

>> You could certainly *drive* the migration process out of SCons, but even there I don't think SCons's dependency analysis will be all that helpful -- I'd just use os.system() to kick off your migration script (or make it be a python function and just execute it straight out of your SConstruct).

> 

> 

> For developer yes, but I would like to use scons also on the userside and than I need an easy call, but not in all system I can use a GUI based package.

> 

> It is only an idea and I would like thinking about it, because the Python structure of Scons can be helpfull.

> 

> Thanks

> 

> Phil

> 

> 

>> 

>> 

>> On Sat, Apr 13, 2013 at 5:34 AM, Philipp Kraus <philipp.kraus at flashpixx.de> wrote:

>> Hi,

>> 

>> I'm searching for database support in Scons something like databse installation and updating. I'm often work with database and do the installation with SQL scripts,

>> but the client is compiled and installed with Scons, so can be this an idea for Scons? Do you think this is a good & meningful feature?

>> At the moment I run env.Command with the database command-line client, but the updating process can not work / can only work with deleting & creating the database.

>> 

>> I think a feature like this

>> env.CreateDatabase("localhost:5432", databse="testdb", user"=myself", password="1234", type="postgresql", source="database.sql.scons")

>> 

>> should be nice, also an env.UpdateDatabase which updates the database without deleting tables, it creates only alter statements and run these statements for the update process.

>> Scons can handle this calls in an transaction for atomic calls and uses Pythons database packages. The database script should be more than a plain text file, so Scons can

>> check the database and create individual statements during the update process.

>> 

>> My idea is, that the Scons database.sql.scons file is a XML files which descriptes tables, indicies, references datatypes (validation via XSD)....

>> On the createdatabase Scons transform this data with XSLT into SQL or walk through the XML tree and creates on-fly the statements and send it over Python Database API

>> The update is more difficult, because I must create a diff between installed tables and new table definition. I think a post- / preprocessing should be also in it, so the user

>> can run some codes before / after the update for modifying datasets.

>> 

>> Hope for some comments :-)

>> Have a nice weekend

>> 

>> Phil

>> _______________________________________________

>> Scons-users mailing list

>> Scons-users at scons.org

>> http://four.pairlist.net/mailman/listinfo/scons-users

>> 

>> 

>> 

>> -- 

>> Gary

>> _______________________________________________

>> Scons-users mailing list

>> Scons-users at scons.org

>> http://four.pairlist.net/mailman/listinfo/scons-users

> 

> 

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users

> 

> 

> _______________________________________________

> Scons-users mailing list

> Scons-users at scons.org

> http://four.pairlist.net/mailman/listinfo/scons-users


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://four.pairlist.net/pipermail/scons-users/attachments/20130413/ddab09c4/attachment.htm>