Spark Query Taking Forever?
Crashing with out-of-memory errors?
Introducing KwikQuery’s TabbyDB: a turbocharged fork of Apache Spark for
lightning-fast queries and unbeatable performance.
KwikQuery Objective
Reduce query delays and improve runtime efficiency.
Query
Compile
Spark UI doesn’t count query compile time, masking potential and bottlenecks.
Planning
Execution
More time may get spent in planning than executing for large, complex queries.
Runtime
Performance
Runtime performance of nested join queries remains unsatisfactory and slow.
UI
Register
The UI only registers a query after the plan is submitted, which can hide bottlenecks.
Work
Around
Some providers suggest disabling optimization rules, but this negatively impacts runtime performance.
UI
Register
UI register outcome includes wasted compute, significant delays, and unmet SLAs.
Accelerating Complex Query Execution
TabbyDB enhances Apache Spark by eliminating the performance bottlenecks, which impact intricate queries (complex case logic, extensive joins, and large query trees), resulting in speed up of execution and reduced resource consumption, enabling enterprises to handle demanding data workloads more efficiently.
Check out the performance difference between stock spark and KwikQuery's TabbyDB, by clicking the button for comparing performance. It will lead you to Zeppelin notebooks, where same query can be run on stock spark and TabbyDB. Please note that running the paragraph on stock spark may take anywhere between 5 to 12 minutes.
Performance Enhancements That Redefine Complex Query Execution
Optimize complex queries for faster, more efficient execution.
Time
Optimization
Changing at fundamental level, the algorithm of some of the critical rules like constraints propagation, collapsing the project nodes early (analysis phase), minimizing the calls to hive meta store & other modifications, tremendously improve compile-time performance.
Advanced
Broadcast
Our fork optimizes the broadcast hash joins on non partitioned columns, to do dynamic file pruning, boosting the runtime performance of nested join queries. In a limited TPCDS testing, it has shown 13% performance improvement in time taken, compared to stock spark.
Caches
Optimization
The cache lookup of in-memory plans is now more intelligent, improving the hit rate of successful lookups and reducing unnecessary computations. This heightened sensitivity can significantly enhance overall runtime performance and efficiency for complex queries.
Scalable
Query
The new rules & algorithm change allow for collapse of projects in the analysis phase, thereby capping the tree size. This results in savings in terms of time taken to compile, preventing out of memory errors.
Seamless
Integration
While boosting performance, KwikQuery's TabbyDB, retains full compatibility with Apache Spark’s APIs and features, allowing users to leverage familiar tools with enhanced speed.
System
Reliability
KwikQuery’s TabbyDB ensures consistent and reliable query execution, even under heavy workloads, while maintaining full Spark compatibility and minimizing unexpected failures.
The Solution: KwikQuery's TabbyDB
- We tackle the root causes:
- Query plan bloat
- Compile-time overhead
- Suboptimal runtime performance of nested join queries
- Absolutely no disabling of rules!
- No code rewrites
- No cluster changes
- Dynamic file pruning using broadcast hash join data, to better run time performance
Frequently asked questions
Find clear answers to common questions about TabbyDB’s capabilities, performance, and integration to help you make the most of our advanced query engine.
TabbyDB is a specialized fork of Apache Spark designed to optimize complex queries. It significantly reduces compilation time and memory usage for queries with nested joins, complex case statements, and large query trees through intelligent compile-time and runtime enhancements.
Yes, TabbyDB is built to manage vast and intricate query structures. Its optimizations improve both the speed and resource consumption, enabling faster execution of queries that would typically take hours to compile and run.
TabbyDB maintains full compatibility with Apache Spark’s DataFrame APIs, allowing corporate users to continue using their programmatic query methods while benefiting from enhanced performance without changing their existing codebase.
Start your free trial
Share your details and we’ll send you a download link shortly.
