Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

For merge #156

Open
wants to merge 64 commits into
base: for_auto_test
Choose a base branch
from
Open

For merge #156

wants to merge 64 commits into from

Conversation

frankliee
Copy link
Contributor

No description provided.

yukai2014 and others added 30 commits April 12, 2016 15:45
@yukai2014
ADD: master loader and slave loader as well as some config variables …
08967d1 
…OPTIMIZE: some flags in Makefile.am
@yukai2014
FIX: name from 'inject' to 'ingest'
a78cbad
@frankliee
txn_manager first commit
00b42f0
@yukai2014
ADD: stl_guard to implement RAII for STL container; ADD: function imp…
202c782 
…lementation in MasterLoader
@yukai2014
Merge remote-tracking branch 'remotes/origin/lizhifang-dLoad-160416' …
8ffb3fc 
…into DistributedLoad
@frankliee
add log to txn_manager
9c2a150
@frankliee
change path of txn_server/client_test
9e7edd0
@frankliee
rm cout
b8e03ae
@frankliee
for cluster test
95605d8
@frankliee
add txn manager and log to config and enviroment
6a09bdd
@yukai2014
complete master loader but GetMessage(); ADD: load packet; ADD: resou…
941616e 
…rce guard to auto-release memory after function returning
@yukai2014
Merge remote-tracking branch 'remotes/origin/lizhifang-dLoad-160421' …
429e63f 
…into DistributedLoad

Conflicts:
	loader/master_loader.cpp
	txn_manager/txn.hpp
	txn_manager/txn_server.cpp
@frankliee
add txnPartitionReadIterator
2801b6d
@frankliee
find a caf actor bug
706393b
@frankliee
refactor txn_manager
2a5097c
@yukai2014
slave loader is to be finished
2ec1376
@frankliee
basiclly implement pipeline of txn manager
78fec97
@frankliee
finish test project
03c13f4
@yukai2014
finish slave loader but failed to compile
06e6b8e
@frankliee
update txn_test
bf4f8d4
@yukai2014
ADD: simple test case; compile ok
e6db4bd
@yukai2014
ADD: some debug info log
23f2c02
@yukai2014
ADD: InitConnector() in table.cpp
bca77d3 
OPTIMIZE: master_loader_actor is not member variable
ERROR version
@yukai2014
network connection ok, but master loader lack node info
a4877ca
@frankliee
init txn-manager from catalog
b74c277
@yukai2014
Completed Loader without MQ! FIX: add bracket around macro definition…
5c6d5ff 
…; FIX: bug found in LoadPacket/SlaveLoader/MasterLoader/ChunkStorage, txn.cpp; ADD: announce custom structure for CAF; ADD: send binding info and node register info to master loader;
@yukai2014
ADD:AMQ consumer in multi-threading; FIX: bug in ChunkStorage.cpp; AD…
e374453 
…D: memset after getting memory from memory pool
@yukai2014
Merge remote-tracking branch 'origin/lizhifang-dLoad-160502' into Dis…
5315ac8 
…tributedLoad

Conflicts:
	Environment.cpp
	loader/master_loader.cpp
	storage/PartitionStorage.h
	txn_manager/txn.hpp
	txn_manager/txn_server.cpp
@yukai2014
Distributed Load basically works; FIX: memory leak by getSchema(); FI…
23c2809 
…X: bug in ChunkStorage.cpp
@yukai2014
Distributed Load in one partition Ok! FIX: use std::unordered_map rat…
7dfad8a 
…her than boost::unordered_map;
yukai2014 and others added 30 commits May 9, 2016 13:48
@yukai2014
ADD: elapsed time for 1000 transactions
c04badd
@yukai2014
OPTIMIZE: doing sending packet in another thread
11336e3
@yukai2014
OPTIMIZE: disable some debug log to get better performance;BUG: some …
8d10e0b 
…txn id are duplicated
@yukai2014
FIX: bug caused by multiple txn_servers and lack of lock in reading c…
c2f8208 
…ommit_info; ADD: switch release/debug in configure.ac; ADD: multi-thread sending with multiple queues
@yukai2014
ADD: performance info output
06a347d
@yukai2014
Great performance improvment! OTPIMIZE: set remote actor as static ac…
6c71cc3 
…tor; slave loader don't wait for master loader's feedback
@yukai2014
ADD: performance log
ae12356
@yukai2014
Ready for measuring performance. ADD: handle work thread based on CAF…
83557cf 
… in slave_loader;
@yukai2014
Performance test version
40268e4
@yukai2014
FIX: using the old partition reader
2067270
@yukai2014
FIX: a bug in InitTxnManager()
805faee
@yukai2014
No complain from compiler; Failed to execute
33d011d
@yukai2014
OPTIMIZE:adjust the order of initialize master_loader and master_node…
365b82e 
…; FIX: bug in catalog.cpp
@yukai2014
ADD: multi-thread slave loader; ADD: concurrency control on storage w…
6b59d84 
…hen multi-thread loading
@yukai2014
ADD: comments about why current scan can't read new data
e2d0ae8
@yukai2014
FIX: bug appeared when txn_log is disable
6e71ec6
@frankliee
Merge branch 'auto_test_ingest' into ingestion_yukai
2977d05 
Conflicts:
	node_manager/slave_node.cpp
	physical_operator/Makefile.am
@frankliee
rm useless txn_manager/log_client.hpp,log_client.cpp,log_server.hpp,l…
b1da537 
…og_server.cpp
@frankliee
implement timestamp and txn binning for ingestion
fd067e0
@frankliee
add txn info for a select query
ca73e30
@frankliee
implement real-time chunk_list for ingest,
5523080 
scan from both real-time chunk_list and historical chunk_list,
advaned binding all partitions in claims initialize stage
@frankliee
implement persist
d6453d8
@frankliee
fix bugs for ingestion and txn-scan(rt chunk list bug)
01b07d1
@frankliee
fix bug for memory merge and persist(don't implement push to HDFS)
5db604e
@frankliee
fix a bug for rt-scan
0934b46
@frankliee
fix a bug for "count (*) " miss lines caused by block tail wrong init
98d14f8
@frankliee
fix bug for memory leak(data ingestion) and chunk memory alloc(level …
b50393d 
…HDFS->MEMORY)
@frankliee
implement data persitence for ingestion
4577613
@frankliee
Merge branch 'ingestion_yukai' into for_merge
54ae08c 
Conflicts:
	Server.cpp
	catalog/catalog.cpp
	catalog/partitioner.h
	catalog/projection_binding.cpp
	conf/config
	node_manager/master_node.cpp
	node_manager/slave_node.cpp
	node_manager/slave_node.h
	stmt_handler/load_exec.cpp
	storage/BlockManager.cpp
	storage/ChunkStorage.cpp
@frankliee
ADD: code comment for txn_manager
7aa2942 
FIX: "LoadFromDisk" should use 64bit int as file cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@frankliee @yukai2014