Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicates missed when mate has low mapping quality #128

Closed
yfarjoun opened this issue Dec 18, 2014 · 13 comments · May be fixed by #1460
Closed

Duplicates missed when mate has low mapping quality #128

yfarjoun opened this issue Dec 18, 2014 · 13 comments · May be fixed by #1460
Assignees
Labels

Comments

@yfarjoun
Copy link
Contributor

When a pair-end read has one read with good mapping quality and the second with low (e.g. 0) mapping quality, the second may be placed in several locations randomly according to the aligner (even with the same aligner). This means that a duplicated such fragment will be incorrectly not marked as duplicate if the two version of the second read are aligned differently (note that this has nothing to do with secondary or supplemental alignments)

Perhaps the solution could be as simple as to add another condition to line 283 in MarkDuplicates.java (and perhaps a similar condition in MarkDuplicatesWithCigar?) that verifies that the mate is well mapped. I suspect that a MQ!=0 would suffice, though it depends on the aligner...so perhaps it could be an @option.

@nh13 nh13 self-assigned this Dec 18, 2014
@nh13
Copy link
Collaborator

nh13 commented Dec 18, 2014

So this may be more challenging than initially thought. I tried a quick attempt here: https://github.com/broadinstitute/picard/tree/nh_mark_duplicates_with_low_q_end

Consider the case if we treat read pairs where both ends map, but one end has low mapping quality (ex. 0). Lets say we treat them as fragments. What happens to the high mapq end of the pair. Well it will be marked as a duplicate if there are other pairs with one end at the same (5' unclipped) position, else it may not be marked as duplicate. This seems reasonable. But if the other end is randomly assigned to one of say 10 equally likely alignment positions (say we have large segmental duplications), then when we treat it as a fragment it may not get duplicate marked. We want them to be consistent between both ends (i.e. both are marked as duplicates or neither). We could treat any end of a pair below the mapping quality threshold as unmapped.

Nonetheless, this will require some rigorous testing, and some serious thought, so I am deferring this until we have some time to devote to it.

@nh13 nh13 assigned yfarjoun and unassigned nh13 Dec 18, 2014
@eitanbanks
Copy link
Contributor

Interesting. So what I think you're saying is that it's going to be hard to remove this class of duplicates given the current implementation (which looks at start/end pos of the paired reads).

Wasn't there an implementation of duplicate marking done by Chris Hartl (under the wing of Mauricio at the time and with Tim F advising) that looked at the sequence context of the reads rather than start/end pos? That should theoretically work really well here. I'll investigate.

@nh13
Copy link
Collaborator

nh13 commented Dec 19, 2014

@eitanbanks I mean to say that it will take more than 30 minutes to implement so we are deferring.

@eitanbanks
Copy link
Contributor

Yes, great. Yossi and I just talked through this and think we've come up with a solution that should work (I'll let him describe at his own leisure).

I want an answer to a question about the Buick data now - that's why I'm looking at the other implementation today.

@yfarjoun
Copy link
Contributor Author

yfarjoun commented Jan 6, 2015

@eitanbanks ran a foghorn implementation of MarkDuplicates tool that uses (a Fourier transform of) the 5' end sequences to identify the fragments. He reports that this resolves the problem described here. This presents two possibilities (as I see it):

  1. Convert Picard to be sequence based rather that alignment based (large change in the way things work, but consistent)
  2. Add an option that when a read has low MQ, the sequence is used rather than the alignment (much less disruptive, but less consistent)

It should be said that this problem presents mainly in low-complexity, high depth samples which are not something we are promoting...so this points to the second solution since it would entail minimal disruption and presumably be less time-consuming.

@bhandsaker
Copy link
Contributor

If you are planning to change the behavior of MarkDuplicates, I would
ideally like the chance to be able to test how it affects CNV calling
(it may improve things).
Any chance you could implement this as an option and, ideally, help me
locate or create a data set (ideally with many samples) with MD run both
ways that I could evaluate?
Also, we should make sure that any bams are clearly marked as to which
method is used.
It is quite possible that mixing bams with the two methodologies in
joint CNV calling could create batch effects that would be difficult to
overcome post-hoc.

-Bob

On 1/6/15, 4:32 PM, Yossi Farjoun wrote:

@eitanbanks https://github.com/eitanbanks ran a foghorn
implementation of MarkDuplicates tool that uses (a Fourier transform
of) the 5' end sequences to identify the fragments. He reports that
this resolves the problem described here. This presents two
possibilities (as I see it):

  1. Convert Picard to be sequence based rather that alignment based
    (large change in the way things work, but consistent)
  2. Add an option that when a read has low MQ, the sequence is used
    rather than the alignment (much less disruptive, but less consistent)

It should be said that this problem presents mainly in low-complexity,
high depth samples which are not something we are promoting...so this
points to the second solution since it would entail minimal disruption
and presumably be less time-consuming.


Reply to this email directly or view it on GitHub
#128 (comment).

@yfarjoun
Copy link
Contributor Author

yfarjoun commented Feb 9, 2015

This issue also seems to come up in samples with high chimeric rates, some
of the reads are unmapped and their mates are not properly marked as
duplicates, though looking at IGV, they seem to be.

The screen shot is from sample
...

Yossi.

On Fri, Dec 19, 2014 at 11:53 AM, Eric Banks [email protected]
wrote:

Yes, great. Yossi and I just talked through this and think we've come up
with a solution that should work (I'll let him describe at his own leisure).

I want an answer to a question about the Buick data now - that's why I'm
looking at the other implementation today.


Reply to this email directly or view it on GitHub
#128 (comment)
.

@nh13
Copy link
Collaborator

nh13 commented Mar 6, 2015

@yfarjoun should we be posting file paths here?

@nh13 nh13 added the bug label Mar 6, 2015
@pmBarlev
Copy link

@yfarjoun I have a bam with two chimeric read pairs in which:

  1. Both reads in both pairs have mapping quality 0.
  2. Both reads in both pairs are aligned the same.
  3. They are not marked as duplicates.

This doesn't quite the fit the bill of the bug you describe above (since both reads do align the same), but seems related. Do you know why this happens - perhaps because there is no mate-cigar tag (why isn't there one, other reads in the bam have it)?

I'm attaching the relevant portion of the .bam:

@PG ID:GATK IndelRealigner  CL:knownAlleles=[] targetIntervals=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/MUTECT_SC_N_VS_C_20151026_B37/GATK/DF109EBF-AEE9-D348-9C72-BF958C37ABB9.RealignTarget.list LODThresholdForCleaning=0.4 consensusDeterminationModel=USE_READS entropyThreshold=0.15 maxReadsInMemory=150000 maxIsizeForMovement=3000 maxPositionalMoveAllowed=200 maxConsensuses=30 maxReadsForConsensuses=120 maxReadsForRealignment=20000 noOriginalAlignmentTags=false nWayOut=null generate_nWayOut_md5s=false check_early=false noPGTag=false keepPGTags=false indelsFileForDebugging=null statisticsFileForDebugging=null SNPsFileForDebugging=null
@PG ID:GATK IndelRealigner.1    CL:knownAlleles=[] targetIntervals=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/MUTECT_SC_N_VS_C_20151026_B37/GATK/2FE1A718-FD4C-0546-B3F7-7FF35964E264.RealignTarget.list LODThresholdForCleaning=0.4 consensusDeterminationModel=USE_READS entropyThreshold=0.15 maxReadsInMemory=150000 maxIsizeForMovement=3000 maxPositionalMoveAllowed=200 maxConsensuses=30 maxReadsForConsensuses=120 maxReadsForRealignment=20000 noOriginalAlignmentTags=false nWayOut=null generate_nWayOut_md5s=false check_early=false noPGTag=false keepPGTags=false indelsFileForDebugging=null statisticsFileForDebugging=null SNPsFileForDebugging=null
@PG ID:GATK IndelRealigner.2    CL:knownAlleles=[] targetIntervals=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/MUTECT_SC_N_VS_C_20151026_B37/GATK/47242324-DECD-3049-ABB4-CEE5FD1EB788.RealignTarget.list LODThresholdForCleaning=0.4 consensusDeterminationModel=USE_READS entropyThreshold=0.15 maxReadsInMemory=150000 maxIsizeForMovement=3000 maxPositionalMoveAllowed=200 maxConsensuses=30 maxReadsForConsensuses=120 maxReadsForRealignment=20000 noOriginalAlignmentTags=false nWayOut=null generate_nWayOut_md5s=false check_early=false noPGTag=false keepPGTags=false indelsFileForDebugging=null statisticsFileForDebugging=null SNPsFileForDebugging=null
@PG ID:GATK IndelRealigner.3    CL:knownAlleles=[] targetIntervals=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/MUTECT_SC_N_VS_C_20151026_B37/GATK/441CF955-45F0-E04C-9761-6F44CE08ECF9.RealignTarget.list LODThresholdForCleaning=0.4 consensusDeterminationModel=USE_READS entropyThreshold=0.15 maxReadsInMemory=150000 maxIsizeForMovement=3000 maxPositionalMoveAllowed=200 maxConsensuses=30 maxReadsForConsensuses=120 maxReadsForRealignment=20000 noOriginalAlignmentTags=false nWayOut=null generate_nWayOut_md5s=false check_early=false noPGTag=false keepPGTags=false indelsFileForDebugging=null statisticsFileForDebugging=null SNPsFileForDebugging=null
@PG ID:GATK PrintReads  VN:3.3-0-g37228af   CL:readGroup=null platform=null number=-1 sample_file=[] sample_name=[] simplify=false no_pg_tag=false
@PG ID:MarkDuplicates   PN:MarkDuplicates   VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010)   CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/bam/SC241_WS4N_GCCAAT_L001.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L001_marked.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L001_metrics.txt    PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@PG ID:MarkDuplicates.5 PN:MarkDuplicates   VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010)   CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/bam/SC241_WS4N_GCCAAT_L002.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L002_marked.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L002_metrics.txt    PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@PG ID:MarkDuplicates.6 PN:MarkDuplicates   VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010)   CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/bam/SC241_WS4N_GCCAAT_L003.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L003_marked.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L003_metrics.txt    PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@PG ID:MarkDuplicates.7 PN:MarkDuplicates   VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010)   CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/bam/SC241_WS4N_GCCAAT_L004.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L004_marked.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L004_metrics.txt    PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@PG ID:MarkDuplicates.9 PN:MarkDuplicates   PP:MarkDuplicates.7 VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010)   CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L001_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L002_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L003_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L004_marked_align_bqsr.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N_metrics.txt    PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@PG ID:MarkDuplicates.A PN:MarkDuplicates   PP:MarkDuplicates.6 VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010)   CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L001_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L002_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L003_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L004_marked_align_bqsr.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N_metrics.txt    PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@PG ID:MarkDuplicates.B PN:MarkDuplicates   PP:MarkDuplicates.5 VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010)   CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L001_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L002_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L003_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L004_marked_align_bqsr.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N_metrics.txt    PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@PG ID:MarkDuplicates.C PN:MarkDuplicates   PP:MarkDuplicates   VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010)   CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L001_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L002_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L003_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L004_marked_align_bqsr.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N_metrics.txt    PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
HISEQ:99:C6W0CACXX:2:2213:16816:19878   337 1   10000   0   52H48M  6   160131968   0   ATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACC    =>==>>>=<<==>=;;===<::==<;::<<=;::==?>====>><;==    SA:Z:6,160132336,-,58M42S,0,0;  BD:Z:IHHJHKKHHJHKKHHJHKKHHJHKKHHJHKKHHKILLIJLKOOLHJGG   PG:Z:MarkDuplicates.B   RG:Z:CutadaptFastq4PPLall_L002  BI:Z:LLKNJLNKJMIKMKJMIKMKJMIKMKJMIKMKJMIKMKKNJMPNKNJJ   NM:i:0  AS:i:48 XS:i:47
HISEQ:99:C6W0CACXX:3:2103:10975:91079   337 1   10000   0   52H48M  6   160131968   0   ATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACC    =<===>>;<<==>;<3=<=:;;88<:::;3<;:;;=?>====>9<;==    SA:Z:6,160132336,-,58M42S,0,0;  BD:Z:IHHJHKKHHJHKKHHJHKKHHJHKKHHJHKKHHKILLIJLKOOLHJGG   PG:Z:MarkDuplicates.A   RG:Z:CutadaptFastq4PPLall_L003  BI:Z:LLKMJKNKJLIKMKJLIKMKJLIKMKJLIKMKJLIKNKKMJMPNKMJJ   NM:i:0  AS:i:48 XS:i:47

@yfarjoun
Copy link
Contributor Author

I see that the records are from different readgroups. Could you include
their @rg tags as well? I want to see if they are from the same library or
not.

Also, could you send the mates? I'd like to see the 5' end of their
alignment.

On Thu, Nov 19, 2015 at 2:24 AM, pmBarlev [email protected] wrote:

@yfarjoun https://github.com/yfarjoun I have a bam with two chimeric
read pairs in which:

  1. Both reads in both pairs have mapping quality 0.
  2. Both reads in both pairs are aligned the same.
  3. They are not marked as duplicates.

This doesn't quite the fit the bill of the bug you describe above (since
both reads do align the same), but seems related. Do you know why this
happens?

I'm attaching the relevant portion of the .bam:

@pg ID:GATK IndelRealigner CL:knownAlleles=[] targetIntervals=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/MUTECT_SC_N_VS_C_20151026_B37/GATK/DF109EBF-AEE9-D348-9C72-BF958C37ABB9.RealignTarget.list LODThresholdForCleaning=0.4 consensusDeterminationModel=USE_READS entropyThreshold=0.15 maxReadsInMemory=150000 maxIsizeForMovement=3000 maxPositionalMoveAllowed=200 maxConsensuses=30 maxReadsForConsensuses=120 maxReadsForRealignment=20000 noOriginalAlignmentTags=false nWayOut=null generate_nWayOut_md5s=false check_early=false noPGTag=false keepPGTags=false indelsFileForDebugging=null statisticsFileForDebugging=null SNPsFileForDebugging=null
@pg ID:GATK IndelRealigner.1 CL:knownAlleles=[] targetIntervals=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/MUTECT_SC_N_VS_C_20151026_B37/GATK/2FE1A718-FD4C-0546-B3F7-7FF35964E264.RealignTarget.list LODThresholdForCleaning=0.4 consensusDeterminationModel=USE_READS entropyThreshold=0.15 maxReadsInMemory=150000 maxIsizeForMovement=3000 maxPositionalMoveAllowed=200 maxConsensuses=30 maxReadsForConsensuses=120 maxReadsForRealignment=20000 noOriginalAlignmentTags=false nWayOut=null generate_nWayOut_md5s=false check_early=false noPGTag=false keepPGTags=false indelsFileForDebugging=null statisticsFileForDebugging=null SNPsFileForDebugging=null
@pg ID:GATK IndelRealigner.2 CL:knownAlleles=[] targetIntervals=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/MUTECT_SC_N_VS_C_20151026_B37/GATK/47242324-DECD-3049-ABB4-CEE5FD1EB788.RealignTarget.list LODThresholdForCleaning=0.4 consensusDeterminationModel=USE_READS entropyThreshold=0.15 maxReadsInMemory=150000 maxIsizeForMovement=3000 maxPositionalMoveAllowed=200 maxConsensuses=30 maxReadsForConsensuses=120 maxReadsForRealignment=20000 noOriginalAlignmentTags=false nWayOut=null generate_nWayOut_md5s=false check_early=false noPGTag=false keepPGTags=false indelsFileForDebugging=null statisticsFileForDebugging=null SNPsFileForDebugging=null
@pg ID:GATK IndelRealigner.3 CL:knownAlleles=[] targetIntervals=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/MUTECT_SC_N_VS_C_20151026_B37/GATK/441CF955-45F0-E04C-9761-6F44CE08ECF9.RealignTarget.list LODThresholdForCleaning=0.4 consensusDeterminationModel=USE_READS entropyThreshold=0.15 maxReadsInMemory=150000 maxIsizeForMovement=3000 maxPositionalMoveAllowed=200 maxConsensuses=30 maxReadsForConsensuses=120 maxReadsForRealignment=20000 noOriginalAlignmentTags=false nWayOut=null generate_nWayOut_md5s=false check_early=false noPGTag=false keepPGTags=false indelsFileForDebugging=null statisticsFileForDebugging=null SNPsFileForDebugging=null
@pg ID:GATK PrintReads VN:3.3-0-g37228af CL:readGroup=null platform=null number=-1 sample_file=[] sample_name=[] simplify=false no_pg_tag=false
@pg ID:MarkDuplicates PN:MarkDuplicates VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010) CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/bam/SC241_WS4N_GCCAAT_L001.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L001_marked.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L001_metrics.txt PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_
LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@pg ID:MarkDuplicates.5 PN:MarkDuplicates VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010) CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/bam/SC241_WS4N_GCCAAT_L002.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L002_marked.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L002_metrics.txt PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_
LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@pg ID:MarkDuplicates.6 PN:MarkDuplicates VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010) CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/bam/SC241_WS4N_GCCAAT_L003.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L003_marked.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L003_metrics.txt PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_
LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@pg ID:MarkDuplicates.7 PN:MarkDuplicates VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010) CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/bam/SC241_WS4N_GCCAAT_L004.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L004_marked.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/markedBam/SC241_WS4N_GCCAAT_L004_metrics.txt PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_
LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@pg ID:MarkDuplicates.9 PN:MarkDuplicates PP:MarkDuplicates.7 VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010) CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L001_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L002_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L003_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L004_marked_align_bqsr.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC24
1_WS4N/SC241_WS4N_metrics.txt PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@pg ID:MarkDuplicates.A PN:MarkDuplicates PP:MarkDuplicates.6 VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010) CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L001_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L002_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L003_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L004_marked_align_bqsr.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC24
1_WS4N/SC241_WS4N_metrics.txt PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@pg ID:MarkDuplicates.B PN:MarkDuplicates PP:MarkDuplicates.5 VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010) CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L001_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L002_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L003_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L004_marked_align_bqsr.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC24
1_WS4N/SC241_WS4N_metrics.txt PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
@pg ID:MarkDuplicates.C PN:MarkDuplicates PP:MarkDuplicates VN:1.115(30b1e546cc4dd80c918e151dbfe46b061e63f315_1402927010) CL:picard.sam.MarkDuplicates INPUT=[/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L001_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L002_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L003_marked_align_bqsr.bam, /bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/CutadaptFastq4PPLall/alignBqsr/SC241_WS4N_GCCAAT_L004_marked_align_bqsr.bam] OUTPUT=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC241_WS4N/SC241_WS4N.bam METRICS_FILE=/bio/projects/Sheba/MayaDadiani/Data/Mutect_SC_N_vs_C_20151026_B37/libs/SC24
1_WS4N/SC241_WS4N_metrics.txt PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates REMOVE_DUPLICATES=false ASSUME_SORTED=false MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
HISEQ:99:C6W0CACXX:2:2213:16816:19878 337 1 10000 0 52H48M 6 160131968 0 ATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACC =>==>>>=<<==>=;;===<::==<;::<<=;::==?>====>><;== SA:Z:6,160132336,-,58M42S,0,0; BD:Z:IHHJHKKHHJHKKHHJHKKHHJHKKHHJHKKHHKILLIJLKOOLHJGG PG:Z:MarkDuplicates.B RG:Z:CutadaptFastq4PPLall_L002 BI:Z:LLKNJLNKJMIKMKJMIKMKJMIKMKJMIKMKJMIKMKKNJMPNKNJJ NM:i:0 AS:i:48 XS:i:47
HISEQ:99:C6W0CACXX:3:2103:10975:91079 337 1 10000 0 52H48M 6 160131968 0 ATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACC =<===>>;<<==>;<3=<=:;;88<:::;3<;:;;=?>====>9<;== SA:Z:6,160132336,-,58M42S,0,0; BD:Z:IHHJHKKHHJHKKHHJHKKHHJHKKHHJHKKHHKILLIJLKOOLHJGG PG:Z:MarkDuplicates.A RG:Z:CutadaptFastq4PPLall_L003 BI:Z:LLKMJKNKJLIKMKJLIKMKJLIKMKJLIKMKJLIKNKKMJMPNKMJJ NM:i:0 AS:i:48 XS:i:47


Reply to this email directly or view it on GitHub
#128 (comment)
.

@pmBarlev
Copy link

Perhaps the issue is that these reads are not primary alignments (see line 148 in MarkDuplicates.java)? Indeed, both the corresponding primary alignments and their mates are marked as duplicates.

In any case, here are the read groups:

@RG ID:CutadaptFastq4PPLall_L001    PL:illumina PU:GCCAAT   LB:SC241_WS4N   SM:SC241_WS4N
@RG ID:CutadaptFastq4PPLall_L002    PL:illumina PU:GCCAAT   LB:SC241_WS4N   SM:SC241_WS4N
@RG ID:CutadaptFastq4PPLall_L003    PL:illumina PU:GCCAAT   LB:SC241_WS4N   SM:SC241_WS4N
@RG ID:CutadaptFastq4PPLall_L004    PL:illumina PU:GCCAAT   LB:SC241_WS4N   SM:SC241_WS4N

Here are the mates:

HISEQ:99:C6W0CACXX:2:2213:16816:19878   1185    6   160131968   0   91M9S   =   160132336   426 CATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCCTAACCCTA    :97::998:;9;:;:;:;;<;;<<=<<====<=>?>>=>>>>?>?@@??>??<>?>=>=>>?>>>>?<<<;<<:;<<<<:<<;<<;;==<==<==<:98;    MC:Z:58M42S BD:Z:GGJIKKLJHJIGIFGGFGIJJJFFJJJJIGJJJIHIJJHJIGJJJIHIJJHJJHJGHHGHJKKKGGKKJJIGJJJIHIJJHJIHKKKJIKLJLMKLJHIJ   PG:Z:MarkDuplicates.B   RG:Z:CutadaptFastq4PPLall_L002  BI:Z:JJMLGJJJHIJHIEHHHIJLLLFILMJJKGILMLHKKLJKKGILMLHKKLJKLJKGJJJJKMMMGJMNJKKGILMLHKKLJKKGIMNMILLLLMJMKJKL   NM:i:0  MQ:i:0  AS:i:91 XS:i:91
HISEQ:99:C6W0CACXX:3:2103:10975:91079   1185    6   160131968   0   91M9S   =   160132336   426 CATAACCACCACTCACAATGCTCTGACCTCCATAACCACCTCCATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCCTAACCCTA    997::8989;9::;:8:*8;99;<=<:<===<==?>>==>>>?>?@>??>=><=>>=>=>>?>>>>?>;<;<<:;:<;;:;;;;;:;:<;;;;:<;8789    MC:Z:58M42S BD:Z:GGJIKKKJHJIGIFGGFGGJJJFFJJIIIGJJJIHIJJHJIGJJJIHIJJHJJHJGHHGHJKKKGGKKJJIGJJJIHIJJHJJHKKKJIJLJLMKLJHIJ   PG:Z:MarkDuplicates.A   RG:Z:CutadaptFastq4PPLall_L003  BI:Z:JJMLGJKJHJJHIEHHIIILLLFILMJKJFILLLHKLLJLKGILMLHKLLJLLJKGJJJJKMMMGJMMJLJGILMLHKLLJLKGIMNMILMKLMJMLJKL   NM:i:1  MQ:i:0  AS:i:86 XS:i:86


And the primary alignments

HISEQ:99:C6W0CACXX:2:2213:16816:19878   1105    6   160132336   0   58M42S  =   160131968   -426    AACCACCTCCATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACC    :97:6<=<<=<2;5<=<<=;>=><?=@<>@>>=?<?@=?@>?>>?@>?@??@=>==>>>=<<==>=;;===<::==<;::<<=;::==?>====>><;==    SA:Z:1,10000,-,52S48M,0,0;  MC:Z:91M9S  BD:Z:HJJHNNMMLMKJJKKIKKILKLIIHKKKLJIKHKJKJJJKIHHJJHJKJJJKIHHJHKKHHJHKKHHJHKKHHJHKKHHJHKKHHKILLIJLKOOLHJGG   PG:Z:MarkDuplicates.B   RG:Z:CutadaptFastq4PPLall_L002  BI:Z:KNLKPNMNNONMLOMLOMLMMOLLLMNNOMKNJNNLLLLMLLKNLKNLLLLMLLKNJLNKJMIKMKJMIKMKJMIKMKJMIKMKJMIKMKKNJMPNKNJJ   NM:i:0  MQ:i:0  AS:i:58 XS:i:58
HISEQ:99:C6W0CACXX:3:2103:10975:91079   1105    6   160132336   0   58M42S  =   160131968   -426    AACCACCTCCATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACC    797:6<=<>=9980<<;:;5<<<67/><;4*==::?@???;)<>?@>>@2??=<===>>;<<==>;<3=<=:;;88<:::;3<;:;;=?>====>9<;==    SA:Z:1,10000,-,52S48M,0,0;  MC:Z:91M9S  BD:Z:HJJHNNMMLMKJIKKIKKILKLIIHKKKLJIKHKJKJJJKIHHJJHJKJJJKIHHJHKKHHJHKKHHJHKKHHJHKKHHJHKKHHKILLIJLKOOLHJGG   PG:Z:MarkDuplicates.A   RG:Z:CutadaptFastq4PPLall_L003  BI:Z:KMLKONMNNONMLNMLNMLMMOLLLMNNOMLNKMNLLMLMLLKMLKMLLLLMLLKMJKNKJLIKMKJLIKMKJLIKMKJLIKMKJLIKNKKMJMPNKMJJ   NM:i:0  MQ:i:0  AS:i:58 XS:i:58

@yfarjoun
Copy link
Contributor Author

That indeed seems to be the issue. (the secondary alignements)

On Thu, Nov 19, 2015 at 7:54 AM, pmBarlev [email protected] wrote:

Perhaps the issue is that these reads are not primary alignments (see line
148 in MarkDuplicates.java)? Indeed, both the corresponding primary
alignments and their mates are marked as duplicates.

In any case, here are the read groups:

@rg ID:CutadaptFastq4PPLall_L001 PL:illumina PU:GCCAAT LB:SC241_WS4N SM:SC241_WS4N
@rg ID:CutadaptFastq4PPLall_L002 PL:illumina PU:GCCAAT LB:SC241_WS4N SM:SC241_WS4N
@rg ID:CutadaptFastq4PPLall_L003 PL:illumina PU:GCCAAT LB:SC241_WS4N SM:SC241_WS4N
@rg ID:CutadaptFastq4PPLall_L004 PL:illumina PU:GCCAAT LB:SC241_WS4N SM:SC241_WS4N

Here are the mates:

HISEQ:99:C6W0CACXX:2:2213:16816:19878 1185 6 160131968 0 91M9S = 160132336 426 CATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCCTAACCCTA :97::998:;9;:;:;:;;<;;<<=<<====<=>?>>=>>>>?>?@@??>??<>?>=>=>>?>>>>?<<<;<<:;<<<<:<<;<<;;==<==<==<:98; MC:Z:58M42S BD:Z:GGJIKKLJHJIGIFGGFGIJJJFFJJJJIGJJJIHIJJHJIGJJJIHIJJHJJHJGHHGHJKKKGGKKJJIGJJJIHIJJHJIHKKKJIKLJLMKLJHIJ PG:Z:MarkDuplicates.B RG:Z:CutadaptFastq4PPLall_L002 BI:Z:JJMLGJJJHIJHIEHHHIJLLLFILMJJKGILMLHKKLJKKGILMLHKKLJKLJKGJJJJKMMMGJMNJKKGILMLHKKLJKKGIMNMILLLLMJMKJKL NM:i:0 MQ:i:0 AS:i:91 XS:i:91
HISEQ:99:C6W0CACXX:3:2103:10975:91079 1185 6 160131968 0 91M9S = 160132336 426 CATAACCACCACTCACAATGCTCTGACCTCCATAACCACCTCCATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCCTAACCCTA 997::8989;9::;:8:*8;99;<=<:<===<==?>>==>>>?>?@>??>=><=>>=>=>>?>>>>?>;<;<<:;:<;;:;;;;;:;:<;;;;:<;8789 MC:Z:58M42S BD:Z:GGJIKKKJHJIGIFGGFGGJJJFFJJIIIGJJJIHIJJHJIGJJJIHIJJHJJHJGHHGHJKKKGGKKJJIGJJJIHIJJHJJHKKKJIJLJLMKLJHIJ PG:Z:MarkDuplicates.A RG:Z:CutadaptFastq4PPLall_L003 BI:Z:JJMLGJKJHJJHIEHHIIILLLFILMJKJFILLLHKLLJLKGILMLHKLLJLLJKGJJJJKMMMGJMMJLJGILMLHKLLJLKGIMNMILMKLMJMLJKL NM:i:1 MQ:i:0 AS:i:86 XS:i:86

And the primary alignments

HISEQ:99:C6W0CACXX:2:2213:16816:19878 1105 6 160132336 0 58M42S = 160131968 -426 AACCACCTCCATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACC :97:6<=<<=<2;5<=<<=;>=>@>>=??>>?@>?@??@=>==>>>=<<==>=;;===<::==<;::<<=;::==?>====>><;== SA:Z:1,10000,-,52S48M,0,0; MC:Z:91M9S BD:Z:HJJHNNMMLMKJJKKIKKILKLIIHKKKLJIKHKJKJJJKIHHJJHJKJJJKIHHJHKKHHJHKKHHJHKKHHJHKKHHJHKKHHKILLIJLKOOLHJGG PG:Z:MarkDuplicates.B RG:Z:CutadaptFastq4PPLall_L002 BI:Z:KNLKPNMNNONMLOMLOMLMMOLLLMNNOMKNJNNLLLLMLLKNLKNLLLLMLLKNJLNKJMIKMKJMIKMKJMIKMKJMIKMKJMIKMKKNJMPNKNJJ NM:i:0 MQ:i:0 AS:i:58 XS:i:58
HISEQ:99:C6W0CACXX:3:2103:10975:91079 1105 6 160132336 0 58M42S = 160131968 -426 AACCACCTCCATAACCACCACTCACACTGCTCTGACCTCCATAACCACCTCCATAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACC 797:6<=<>=9980<<;:;5<<<67/><;4*==::?@???;)<>?@>>@2??=<===>>;<<==>;<3=<=:;;88<:::;3<;:;;=?>====>9<;== SA:Z:1,10000,-,52S48M,0,0; MC:Z:91M9S BD:Z:HJJHNNMMLMKJIKKIKKILKLIIHKKKLJIKHKJKJJJKIHHJJHJKJJJKIHHJHKKHHJHKKHHJHKKHHJHKKHHJHKKHHKILLIJLKOOLHJGG PG:Z:MarkDuplicates.A RG:Z:CutadaptFastq4PPLall_L003 BI:Z:KMLKONMNNONMLNMLNMLMMOLLLMNNOMLNKMNLLMLMLLKMLKMLLLLMLLKMJKNKJLIKMKJLIKMKJLIKMKJLIKMKJLIKNKKMJMPNKMJJ NM:i:0 MQ:i:0 AS:i:58 XS:i:58


Reply to this email directly or view it on GitHub
#128 (comment)
.

@vdauwera
Copy link
Contributor

This thread being well over a year, I'm closing it. Resurrect it if this is still a thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
6 participants