Skip to main content

Optimizing HIVE -BIG DATA

 a. Replacing MR with Tez:-Tez offer API to handle petabytes of data from  across clusters.

b. Follow ORC file format for great performance.

c. Partiting:-Right partitioning based on  logical requirement of data.

d. Bucketing:-Next level of partitioning is more useful when lots of other requirments comes into picture to use same data. Bucketing allow user to use data as per requirement.

e.Vectorization:-One must do the right logical reading by scan,Agg,Filter,Join.

f.CBO-Make sure you do analysis of resource usage before finilizing your parallelism.Check Cost of each process and compare it.

g.Indexing:- Make sure your tables are indexed.

Comments

Post a Comment

Popular posts from this blog

Creating As-Of Versions

To create an As-Of version: 1. From Browse, select a normal version. 2. Select Versions, then Create As-Of Version. 3. Select the As-Of Type: Transaction Date/Time, Transaction Date Offset (Days), or Transaction ID. 4. Perform an action: • For Transaction Date/Time, enter the date and time for the snapshot and click OK. • For Transaction Date Offset (Days), enter the number of days offset from the current date and the time of day and then click OK. • For Transaction ID, enter the transaction ID and click OK, It is Upstream file or load number , DRM just create version for us with this number, It will be useful when downstream want to refer its specific version. When the version tree refreshes, the As-Of version is displayed as a child of its normal version.
Post a Comment
Read more

DRM DRG PROCESS

It explains who did what and when, So that there can be one copy of master data. Any one who is authorized can do that task and can be responsible to that task. we can also figure out what some one has done and  what is pending from his end.
Post a Comment
Read more

epbcs(Version:21.01.34) -application is in the administration mode. new logins are not allowed

 Home>Application->Settings In System Setting you will see Enable use of Application for (Select "All Users") .  From configuration , Refresh your application and it will allow new user to login into EPBCS.
Post a Comment
Read more