BigQuery is amazing. It is one of my favorite tools within Google Cloud. Luckily, it looks like Google feels the same and, to the joy of BigQuery fans, keeps adding new features there. By Artem Nikulchenko.
Let’s say you push some data into BigQuery, and then another system wants to run a scheduled job to process the newly arrived data. For example, a system can try to pull data from BigQuery to another storage, or this system needs to run hourly reports based on the data, etc. In each of those cases, you would prefer to avoid processing the same records multiple times . As a result, you need a way to know which records are already processed and which were added after the processing took place.
No matter how long I have been working with BigQuery, there is always something new I discover once in a while. Today author wants to share with you the following four things:
- AUTO column
- Multi-statement transactions
- Clustering
- Indexes
As you may guess from the name, it is designed for point lookups, but not over any field. Currently, indexes can be used to easily find unique data elements that are buried in unstructured text or semi-structured JSON data. Indexes are only used when the SEARCH query is executed. Good read!
[Read More]