多任务学习早期的研究工作源于对机器学习中的一个重要问题，即“归纳偏置(inductive bias)”问题的研究。机器学习的过程可以看作是对与问题相关的经验数据进行分析，从中归纳出反映问题本质的模型的过程。归纳偏置的作用就是用于指导学习算法如何在模型空间中进行搜索，搜索所得模型的性能优劣将直接受到归纳偏置的影响，而任何一个缺乏归纳偏置的学习系统都不可能进行有效的学习。不同的学习算法(如决策树，神经网络，支持向量机等)具有不同的归纳偏置，人们在解决实际问题时需要人工地确定采用何种学习算法，实际上也就是主观地选择了不同的归纳偏置策略。一个很直观的想法就是，是否可以将归纳偏置的确定过程也通过学习过程来自动地完成，也就是采用“学习如何去学(learning to learn)”的思想。多任务学习恰恰为上述思想的实现提供了一条可行途径，即利用相关任务中所包含的有用信息，为所关注任务的学习提供更强的归纳偏置。
C. Multitasksparsity via maximum entropy discrimination
这篇文章可以看作是比较全面的总结性文章，文中总共讨论了四种情况，feature selection, kernel selection,adaptive pooling and graphical model structure。并详细介绍了四种多任务学习方法，很具有参考价值。
本文只是对多任务学习做了简单的介绍，想要深入了解的话，建议大家看看上面提到的论文。
Reference
[1] T. Evgeniouand M. Pontil. Regularized multi-task learning. In PRoceeding of thetenth ACM SIGKDD international conference on Knowledge Discovery and DataMining, 2004.
[2] T. Jebara. MultitaskSparsity via Maximum Entropy Discrimination. In Journal of Machine LearningResearch, (12):75-110, 2011.
[3] A. Argyriou,T. Evgeniou and M. Pontil. Convex multitask feature learning. In MachineLearning, 73(3):243-272, 2008.
中国科学技术大学多媒体计算与通信教育部-微软重点实验室 MultiMedia Computing Group
Multi-task learning (MTL) is a subfield of machine learning in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately.[1][2][3]
In a widely cited 1997 paper, Rich Caruana gave the following characterization:
Multitask Learning is an approach to inductive transfer that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias. It does this by learning tasks in parallel while using a shared representation; what is learned for each task can help other tasks be learned better.[3]
In the classification context, MTL aims to improve the performance of multiple classification tasks by learning them jointly. One example is a spam-filter, which can be treated as distinct but related classification tasks across different users. To make this more concrete, consider that different people have different distributions of features which distinguish spam emails from legitimate ones, for example an English speaker may find that all emails in Russian are spam, not so for Russian speakers. Yet there is a definite commonality in this classification task across users, for example one common feature might be text related to money transfer. Solving each user's spam classification problem jointly via MTL can let the solutions inform each other and improve performance.[4] Further examples of settings for MTL include multiclass classification and multi-label classification.[5]
Multi-task learning works because regularization induced by requiring an algorithm to perform well on a related task can be superior to regularization that prevents overfitting by penalizing all complexity uniformly. One situation where MTL may be particularly helpful is if the tasks share significant commonalities and are generally slightly under sampled.[4] However, as discussed below, MTL has also been shown to be beneficial for learning unrelated tasks.[6]
Contents
[hide] 1Methods1.1Task grouping and overlap1.2Exploiting unrelated tasks1.3Transfer of knowledge2Mathematics2.1Reproducing Hilbert space of vector valued functions (RKHSvv)2.1.1RKHSvv concepts2.1.2Separable kernels2.1.3Known task structure2.1.3.1Task structure representations2.1.3.2Task structure examples2.1.4Learning tasks together with their structure2.1.4.1Optimization of Q2.1.4.2Special cases2.1.4.3Generalizations3applications3.1Spam filtering3.2Web search3.3RoboEarth4Software package5See also6References7External links7.1Software
Methods[edit]
Task grouping and overlap[edit]
Within the MTL paradigm, information can be shared across some or all of the tasks. Depending on the structure of task relatedness, one may want to share information selectively across the tasks. For example, tasks may be grouped or exist in a hierarchy, or be related according to some general metric. Suppose, as developed more formally below, that the parameter vector modeling each task is a linear combination of some underlying basis. Similarity in terms of this basis can indicate the relatedness of the tasks. For example with sparsity, overlap of nonzero coefficients across tasks indicates commonality. A task grouping then corresponds to those tasks lying in a subspace generated by some subset of basis elements, where tasks in different groups may be disjoint or overlap arbitrarily in terms of their bases.[7] Task relatedness can be imposed a priori or learned from the data.[5][8]
Exploiting unrelated tasks[edit]
One can attempt learning a group of principal tasks using a group of auxiliary tasks, unrelated to the principal ones. In many applications, joint learning of unrelated tasks which use the same input data can be beneficial. The reason is that prior knowledge about task relatedness can lead to sparser and more informative representations for each task grouping, essentially by screening out idiosyncrasies of the data distribution. Novel methods which builds on a prior multitask methodology by favoring a shared low-dimensional representation within each task grouping have been proposed. The programmer can impose a penalty on tasks from different groups which encourages the two representations to be orthogonal. Experiments on synthetic and real data have indicated that incorporating unrelated tasks can result in significant improvements over standard multi-task learning methods.[6]
Transfer of knowledge[edit]
Related to multi-task learning is the concept of knowledge transfer. Whereas traditional multi-task learning implies that a shared representation is developed concurrently across tasks, transfer of knowledge implies a sequentially shared representation. Large scale machine learning projects such as the deep convolutional neural network GoogLeNet,[9] an image-based object classifier, can develop robust representations which may be useful to further algorithms learning related tasks. For example, the pre-trained model can be used as a feature extractor to perform pre-processing for another learning algorithm. Or the pre-trained model can be used to initialize a model with similar architecture which is then fine-tuned to learn a different classification task.[10]
Mathematics[edit]
Reproducing Hilbert space of vector valued functions (RKHSvv)[edit]
The MTL problem can be cast within the context of RKHSvv (a complete inner product space of vector-valued functions equipped with a reproducing kernel). In particular, recent focus has been on cases where task structure can be identified via a separable kernel, described below. The presentation here derives from Ciliberto et al, 2015.[5]
RKHSvv concepts[edit]
Suppose the training data set is {/displaystyle {/mathcal {S}}_{t}=/{(x_{i}^{t},y_{i}^{t})/}_{i=1}^{n_{t}}},%20with {/displaystyle%20x_{i}^{t}/in%20{/mathcal%20{X}}}, {/displaystyle%20y_{i}^{t}/in%20{/mathcal%20{Y}}},%20where {/displaystyle%20t} indexes%20task,%20and {/displaystyle%20t/in%201,...,T}.%20Let {/displaystyle%20n=/sum%20_{t=1}^{T}n_{t}}.%20In%20this%20setting%20there%20is%20a%20consistent%20input%20and%20output%20space%20and%20the%20same loss%20function {/displaystyle%20{/mathcal%20{L}}:/mathbb%20{R}%20/times%20/mathbb%20{R}%20/rightarrow%20/mathbb%20{R}%20_{+}} for%20each%20task:%20.%20This%20results%20in%20the%20regularized%20machine%20learning%20problem:
Choice%20of {/displaystyle%20F:S_{+}^{T}/rightarrow%20/mathbb%20{R}%20_{+}} must%20be%20designed%20to%20learn%20matrices A of%20a%20given%20type.%20See%20"Special%20cases"%20below.Optimization%20of Q[edit]Restricting%20to%20the%20case%20of convex losses%20and coercive penalties%20Ciliberto et%20al have%20shown%20that%20although Q is%20not%20convex%20jointly%20in C and A, a%20related%20problem%20is%20jointly%20convex.Specifically%20on%20the%20convex%20set {/displaystyle%20{/mathcal%20{C}}=/{(C,A)/in%20/mathbb%20{R}%20^{n/times%20T}/times%20S_{+}^{T}|Range(C^{/top%20}KC)/subseteq%20Range(A)/}},%20the%20equivalent%20problem
is%20convex%20with%20the%20same%20minimum%20value.%20And%20if {/displaystyle%20(C_{R},A_{R})} is%20a%20minimizer%20for R then {/displaystyle%20(C_{R}A_{R}^{/dagger%20},A_{R})} is%20a%20minimizer%20for Q.R may%20be%20solved%20by%20a%20barrier%20method%20on%20a%20closed%20set%20by%20introducing%20the%20following%20perturbation:
The%20perturbation%20via%20the%20barrier {/displaystyle%20/delta%20^{2}tr(A^{/dagger%20})} forces%20the%20objective%20functions%20to%20be%20equal%20to {/displaystyle%20+/infty%20} on%20the%20boundary%20of {/displaystyle%20R^{n/times%20T}/times%20S_{+}^{T}} .S can%20be%20solved%20with%20a%20block%20coordinate%20descent%20method,%20alternating%20in C and A. This%20results%20in%20a%20sequence%20of%20minimizers {/displaystyle%20(C_{m},A_{m})} in S that%20converges%20to%20the%20solution%20in R as {/displaystyle%20/delta%20_{m}/rightarrow%200},%20and%20hence%20gives%20the%20solution%20to Q.Special%20cases[edit]Spectral%20penalties -%20Dinnuzo et%20al[11] suggested%20setting F as%20the%20Frobenius%20norm {/displaystyle%20{/sqrt%20{tr(A^{/top%20}A)}}}.%20They%20optimized Q directly%20using%20block%20coordinate%20descent,%20not%20accounting%20for%20difficulties%20at%20the%20boundary%20of {/displaystyle%20/mathbb%20{R}%20^{n/times%20T}/times%20S_{+}^{T}}.Clustered%20tasks%20learning -%20Jacob et%20al[12] suggested%20to%20learn A in%20the%20setting%20where T tasks%20are%20organized%20in R disjoint%20clusters.%20In%20this%20case%20let {/displaystyle%20E/in%20/{0,1/}^{T/times%20R}} be%20the%20matrix%20with {/displaystyle%20E_{t,r}=/mathbb%20{I}%20({/text{task%20}}t/in%20{/text{group%20}}r)}.%20Setting {/displaystyle%20M=I-E^{/dagger%20}E^{T}},%20and{/displaystyle%20U={/frac%20{1}{T}}{/mathbf%20{11}}^{/top%20}},%20the%20task%20matrix {/displaystyle%20A^{/dagger%20}} can%20be%20parameterized%20as%20a%20function%20of {/displaystyle%20M}: {/displaystyle%20A^{/dagger%20}(M)=/epsilon%20_{M}U+/epsilon%20_{B}(M-U)+/epsilon%20(I-M)} ,%20with%20terms%20that%20penalize%20the%20average,%20between%20clusters%20variance%20and%20within%20clusters%20variance%20respectively%20of%20the%20task%20predictions.%20M%20is%20not%20convex,%20but%20there%20is%20a%20convex%20relaxation {/displaystyle%20{/mathcal%20{S}}_{c}=/{M/in%20S_{+}^{T}:I-M/in%20S_{+}^{T}/land%20tr(M)=r/}}.%20In%20this%20formulation, {/displaystyle%20F(A)=/mathbb%20{I}%20(A(M)/in%20/{A:M/in%20{/mathcal%20{S}}_{C}/})}.Generalizations[edit]Non-convex%20penalties -%20Penalties%20can%20be%20constructed%20such%20that%20A%20is%20constrained%20to%20be%20a%20graph%20Laplacian,%20or%20that%20A%20has%20low%20rank%20factorization.%20However%20these%20penalties%20are%20not%20convex,%20and%20the%20analysis%20of%20the%20barrier%20method%20proposed%20by%20Ciliberto%20et%20al%20does%20not%20go%20through%20in%20these%20cases.Non-separable%20kernels -%20Separable%20kernels%20are%20limited,%20in%20particular%20they%20do%20not%20account%20for%20structures%20in%20the%20interaction%20space%20between%20the%20input%20and%20output%20domains%20jointly.%20Future%20work%20is%20needed%20to%20develop%20models%20for%20these%20kernels.Applications[edit]Spam%20filtering[edit]Using%20the%20principles%20of%20MTL,%20techniques%20for%20collaborative spam%20filtering that%20facilitates%20personalization%20have%20been%20proposed.%20In%20large%20scale%20open%20membership%20email%20systems,%20most%20users%20do%20not%20label%20enough%20messages%20for%20an%20individual%20local classifier to%20be%20effective,%20while%20the%20data%20is%20too%20noisy%20to%20be%20used%20for%20a%20global%20filter%20across%20all%20users.%20A%20hybrid%20global/individual%20classifier%20can%20be%20effective%20at%20absorbing%20the%20influence%20of%20users%20who%20label%20emails%20very%20diligently%20from%20the%20general%20public.%20This%20can%20be%20accomplished%20while%20still%20providing%20sufficient%20quality%20to%20users%20with%20few%20labeled%20instances.[13]Web%20search[edit]Using%20boosted decision%20trees,%20one%20can%20enable%20implicit%20data%20sharing%20and%20regularization.%20This%20learning%20method%20can%20be%20used%20on%20web-search%20ranking%20data%20sets.%20One%20example%20is%20to%20use%20ranking%20data%20sets%20from%20several%20countries.%20Here,%20multitask%20learning%20is%20particularly%20helpful%20as%20data%20sets%20from%20different%20countries%20vary%20largely%20in%20size%20because%20of%20the%20cost%20of%20editorial%20judgments.%20It%20has%20been%20demonstrated%20that%20learning%20various%20tasks%20jointly%20can%20lead%20to%20significant%20improvements%20in%20performance%20with%20surprising%20reliability.[14]RoboEarth[edit]In%20order%20to%20facilitate%20transfer%20of%20knowledge,%20IT%20infrastructure%20is%20being%20developed.%20One%20such%20project,%20RoboEarth,%20aims%20to%20set%20up%20an%20open%20source%20internet%20database%20that%20can%20be%20accessed%20and%20continually%20updated%20from%20around%20the%20world.%20The%20goal%20is%20to%20facilitate%20a%20cloud-based%20interactive%20knowledge%20base,%20accessible%20to%20technology%20companies%20and%20academic%20institutions,%20which%20can%20enhance%20the%20sensing,%20acting%20and%20learning%20capabilities%20of%20robots%20and%20other%20artificial%20intelligence%20agents.[15]Software%20package[edit]The%20Multi-Task%20Learning%20via%20StructurAl%20Regularization%20(MALSAR)%20Matlab%20package [16] implements%20the%20following%20multi-task%20learning%20algorithms:Mean-Regularized%20Multi-Task%20Learning[17][18]Multi-Task%20Learning%20with%20Joint%20Feature%20Selection[19]Robust%20Multi-Task%20Feature%20Learning[20]Trace-Norm%20Regularized%20Multi-Task%20Learning[21]Alternating%20Structural%20Optimization[22][23]Incoherent%20Low-Rank%20and%20Sparse%20Learning[24]Robust%20Low-Rank%20Multi-Task%20LearningClustered%20Multi-Task%20Learning[25][26]Multi-Task%20Learning%20with%20Graph%20StructuresSee%20also[edit]Artificial%20IntelligenceArtificial%20neural%20networkEvolutionary%20computationHuman-based%20genetic%20algorithmKernel%20methods%20for%20vector%20outputMachine%20LearningRoboticsReferences[edit]Jump%20up^ Baxter,%20J.%20(2000).%20A%20model%20of%20inductive%20bias%20learning.%20Journal%20of%20Artificial%20Intelligence%20Research,%2012:149--198, On-line%20paperJump%20up^ Thrun,%20S. (1996).%20Is%20learning%20the%20n-th%20thing%20any%20easier%20than%20learning%20the%20first?.%20In%20Advances%20in%20Neural%20Information%20Processing%20Systems%208,%20pp.%20640--646.%20MIT%20Press. Paper%20at%20Citeseer^ Jump%20up%20to:abCaruana,%20R.%20(1997). "Multi-task%20learning" (PDF). Machine%20Learning. doi:10.1023/A:1007379606734.^ Jump%20up%20to:abWeinberger,%20Kilian. "Multi-task%20Learning".^ Jump%20up%20to:abcCiliberto,%20C.%20(2015).%20"Convex%20Learning%20of%20Multiple%20Tasks%20and%20their%20Structure". arXiv:1504.03101.^ Jump up to:ab Romera-Paredes, B., Argyriou, A., Bianchi-Berthouze, N., & Pontil, M., (2012) Exploiting Unrelated Tasks in Multi-Task Learning. http://jmlr.csail.mit.edu/proceedings/papers/v22/romera12/romera12.pdfJump up^ Kumar, A., & Daume III, H., (2012) Learning Task Grouping and Overlap in Multi-Task Learning. http://icml.cc/2012/papers/690.pdfJump up^ Jawanpuria, P., & Saketha Nath, J., (2012) A Convex Feature Learning Formulation for Latent Task Structure Discovery. http://icml.cc/2012/papers/90.pdfJump up^ Szegedy, C. (2014). "Going Deeper with Convolutions". Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. arXiv:1409.4842. doi:10.1109/CVPR.2015.7298594.Jump up^ Roig, Gemma. "Deep Learning Overview" (PDF).Jump up^ Dinuzzo, Francesco (2011). "Learning output kernels with block coordinate descent.". Proceedings of the 28th International Conference on Machine Learning (ICML-11).Jump up^ Jacob, Laurent (2009). "Clustered multi-task learning: A convex formulation". Advances in neural information processing systems.Jump up^ Attenberg, J., Weinberger, K., & Dasgupta, A. Collaborative Email-Spam Filtering with the Hashing-Trick. http://www.cse.wustl.edu/~kilian/papers/ceas2009-paper-11.pdfJump up^ Chappelle, O., Shivaswamy, P., & Vadrevu, S. Multi-Task Learning for Boosting with Application to Web Search Ranking. http://www.cse.wustl.edu/~kilian/papers/multiboost2010.pdfJump up^ Description of RoboEarth ProjectJump up^ Zhou, J., Chen, J. and Ye, J. MALSAR: Multi-tAsk Learning via StructurAl Regularization. Arizona State University, 2012. http://www.public.asu.edu/~jye02/Software/MALSAR. On-line manualJump up^ Evgeniou, T., & Pontil, M. (2004). Regularized multi–task learning. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 109–117).Jump up^ Evgeniou, T., Micchelli, C., & Pontil, M. (2005). Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6, 615.Jump up^ Argyriou, A., Evgeniou, T., & Pontil, M. (2008a). Convex multi-task feature learning. Machine Learning, 73, 243–272.Jump up^ Chen, J., Zhou, J., & Ye, J. (2011). Integrating low-rank and group-sparse structures for robust multi-task learning. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining.Jump up^ Ji, S., & Ye, J. (2009). An accelerated gradient method for trace norm minimization. Proceedings of the 26th Annual International Conference on Machine Learning (pp. 457–464).Jump up^ Ando, R., & Zhang, T. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. The Journal of Machine Learning Research, 6, 1817–1853.Jump up^ Chen, J., Tang, L., Liu, J., & Ye, J. (2009). A convex formulation for learning shared structures from multiple tasks. Proceedings of the 26th Annual International Conference on Machine Learning (pp. 137–144).Jump up^ Chen, J., Liu, J., & Ye, J. (2010). Learning incoherent sparse and low-rank patterns from multiple tasks. Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 1179–1188).Jump up^ Jacob, L., Bach, F., & Vert, J. (2008). Clustered multi-task learning: A convex formulation. Advances in Neural Information Processing Systems， 2008Jump up^ Zhou, J., Chen, J., & Ye, J. (2011). Clustered multi-task learning via alternating structure optimization. Advances in Neural Information Processing Systems.
机器学习知识库
分类：
. doi:10.1109/CVPR.2015.7298594.Jump up^ Roig, Gemma. "Deep Learning Overview" (PDF).Jump up^ Dinuzzo, Francesco (2011). "Learning output kernels with block coordinate descent.". Proceedings of the 28th International Conference on Machine Learning (ICML-11).Jump up^ Jacob, Laurent (2009). "Clustered multi-task learning: A convex formulation". Advances in neural information processing systems.Jump up^ Attenberg, J., Weinberger, K., & Dasgupta, A. Collaborative Email-Spam Filtering with the Hashing-Trick. http://www.cse.wustl.edu/~kilian/papers/ceas2009-paper-11.pdfJump up^ Chappelle, O., Shivaswamy, P., & Vadrevu, S. Multi-Task Learning for Boosting with Application to Web Search Ranking. http://www.cse.wustl.edu/~kilian/papers/multiboost2010.pdfJump up^ Description of RoboEarth ProjectJump up^ Zhou, J., Chen, J. and Ye, J. MALSAR: Multi-tAsk Learning via StructurAl Regularization. Arizona State University, 2012. http://www.public.asu.edu/~jye02/Software/MALSAR. On-line manualJump up^ Evgeniou, T., & Pontil, M. (2004). Regularized multi–task learning. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 109–117).Jump up^ Evgeniou, T., Micchelli, C., & Pontil, M. (2005). Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6, 615.Jump up^ Argyriou, A., Evgeniou, T., & Pontil, M. (2008a). Convex multi-task feature learning. Machine Learning, 73, 243–272.Jump up^ Chen, J., Zhou, J., & Ye, J. (2011). Integrating low-rank and group-sparse structures for robust multi-task learning. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining.Jump up^ Ji, S., & Ye, J. (2009). An accelerated gradient method for trace norm minimization. Proceedings of the 26th Annual International Conference on Machine Learning (pp. 457–464).Jump up^ Ando, R., & Zhang, T. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. The Journal of Machine Learning Research, 6, 1817–1853.Jump up^ Chen, J., Tang, L., Liu, J., & Ye, J. (2009). A convex formulation for learning shared structures from multiple tasks. Proceedings of the 26th Annual International Conference on Machine Learning (pp. 137–144).Jump up^ Chen, J., Liu, J., & Ye, J. (2010). Learning incoherent sparse and low-rank patterns from multiple tasks. Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 1179–1188).Jump up^ Jacob, L., Bach, F., & Vert, J. (2008). Clustered multi-task learning: A convex formulation. Advances in Neural Information Processing Systems， 2008Jump up^ Zhou, J., Chen, J., & Ye, J. (2011). Clustered multi-task learning via alternating structure optimization. Advances in Neural Information Processing Systems.
