Document database backed by keyvalue storage P&L: -7 (≃ -7 HUR)
I am interested in database technology and distributed systems.
I have an experimental python project which is less than 1000 lines and supports basic SQL and SQL inner joins and basic Cypher graph database queries and keyvalue storage. There is no persistence layer.
On the linked website page from this project which is GitHub issues I am talking how I plan to implement document storage so that JSON documents can be mapped to individual keys and joined with SQL queries.
I spent a few hours working on the remainder of this problem and got documnent retrieval working after finishing save document.
This lets you save and retrieve JSON documents
The document data inserted is also queryable by SQL.
Joins against documents are not yet supported but I plan to implement this.
I managed to store a JSON document and I used the SQL inserter to insert the document. In theory the object is queryable by SQL
{ "items": [("name": "item1"), ("name": "item2")], "subobject": {"subobject_key": "value"} }
19,211 2:01
⬜️ need to map a JSON document to numbers so "document id 0 hobbies[0].name=sam" becomes 0@0.1=sam" this allows for efficient retrievals by scanning a range of keys between numbers. ⬜️need to write code to turn numbers back into field names ⬜️need to integrate keyspace optimiser, as we reorder numbers for efficiency of joins across lists for example document 0 = {"hobbies":{("name": "God"),{"name": "databases"), {"name": "computers")}) hobbies might be 0, name might be 1 and we have 3 list items 0, 1, 2 we want all list indexes of the same kind to be adjacent in the sort iterator, so we flip them to the end so 0@0[1]="God" becomes 0@1.0, 0@1.1 0@1.2 ⬜️ need to create keyvalues for what is used for SQL joins ⬜️ need to create a document which defines the joins that people expect to do on a JSON document
19,127 1:02
I am currently in design phase. I am looking for an efficient approach to structured joins and scanning matching keys of collections efficiently.
Great you starting a project, [chronological]! Very interesting. I'll have a deep dive into your thoughts of implementation and code. Perhaps we can also invite here people interested or working on related technology.